{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing Recommender System with SageMaker Built-In Algorithm\n",
    "_**Making Product Recommendations Using Factorization Machines**_\n",
    "\n",
    "--- \n",
    "\n",
    "*This work is based on content from [Gluon based Recommender System notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/gluon_recommender_system/gluon_recommender_system.ipynb)*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "  1. [Explore](#Explore)\n",
    "  1. [Clean](#Clean)\n",
    "  1. [Prepare](#Prepare)\n",
    "1. [Model Training](#Model-Training)\n",
    "1. [Model Inference](#Model-Inference)\n",
    "  1. [Real-Time Inference](#Real-Time-Inference)\n",
    "  1. [Batch Inference](#Batch-Inference)\n",
    "1. [Evaluate Model Performance](#Evaluate-Model-Performance)\n",
    "1. [Model Tuning](#Model-Tuning)\n",
    "1. [Wrap-up](#Wrap-up)\n",
    "  1. [Clean-Up](#Clean-up-(optional))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "In many ways, recommender systems were a catalyst for the current popularity of machine learning.  One of Amazon's earliest successes was the \"Customers who bought this, also bought...\" feature, while the million dollar Netflix Prize spurred research, raised public awareness, and inspired numerous other data science competitions.\n",
    "\n",
    "Recommender systems can utilize a multitude of data sources and ML algorithms, and most combine various unsupervised, supervised, and reinforcement learning techniques into a holistic framework.  However, the core component is almost always a model which predicts a user's rating (or purchase) for a certain item based on that user's historical ratings of similar items as well as the behavior of other similar users.  The minimal required dataset for this is a history of user item ratings.  In our case, we'll use 1 to 5 star ratings from over 2M Amazon customers on over 160K digital videos. More details on this dataset can be found at its [AWS Public Datasets page](https://s3.amazonaws.com/amazon-reviews-pds/readme.html).\n",
    "\n",
    "Matrix factorization has been the cornerstone of most user-item prediction models.  This method starts with the large, sparse, user-item ratings in a single matrix, where users index the rows, and items index the columns.  It then seeks to find two lower-dimensional, dense matrices which, when multiplied together, preserve the information and relationships in the larger matrix.\n",
    "\n",
    "![image](./factorization.png)\n",
    "\n",
    "Matrix factorization has been extended and generalized with deep learning and embeddings.  These techniques allows us to introduce non-linearities for enhanced performance and flexibility.  This notebook will fit a neural network-based model to generate recommendations for the Amazon video dataset.  It will start by exploring our data in the notebook, training a model on the data and fit our model using a SageMaker managed training cluster.  We'll then deploy to an endpoint and check our method.\n",
    "\n",
    "We will also see how the tasks in the machine learning pipeline can be orchestrated and automated through Apache Airflow integration with Sagemaker.\n",
    "\n",
    "---\n",
    "\n",
    "## Setup\n",
    "\n",
    "_This notebook was created and tested on an ml.t2.xlarge notebook instance._\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.  Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the `get_execution_role()` call with the appropriate full IAM role arn string(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handlers could be found for logger \"sagemaker\"\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "prefix = 'sagemaker/fm-recsys'\n",
    "\n",
    "import sagemaker\n",
    "\n",
    "from sagemaker.tuner import HyperparameterTuner, ContinuousParameter\n",
    "from sagemaker.analytics import HyperparameterTuningJobAnalytics, TrainingJobAnalytics\n",
    "\n",
    "#role = sagemaker.get_execution_role()\n",
    "#sess = sagemaker.Session()\n",
    "#smclient = boto3.Session().client('sagemaker')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load the Python libraries we'll need for the remainder of this example notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.sparse import lil_matrix\n",
    "\n",
    "import boto3\n",
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker.predictor import json_deserializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot aesthetics\n",
    "sns.set(color_codes=True)\n",
    "sns.set_context('paper')\n",
    "five_thirty_eight = [\"#30a2da\", \"#fc4f30\", \"#e5ae38\", \"#6d904f\", \"#8b8b8b\",]\n",
    "sns.set_palette(five_thirty_eight)\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Data\n",
    "\n",
    "### Explore\n",
    "\n",
    "Let's start by bringing in our dataset from an S3 public bucket.  As mentioned above, this contains 1 to 5 star ratings from over 2M Amazon customers on over 160K digital videos.  More details on this dataset can be found at its [AWS Public Datasets page](https://s3.amazonaws.com/amazon-reviews-pds/readme.html).\n",
    "\n",
    "_Note, because this dataset is over a half gigabyte, the load from S3 may take ~10 minutes.  Also, since Amazon SageMaker Notebooks start with a 5GB persistent volume by default, and we don't need to keep this data on our instance for long, we'll bring it to the temporary volume (which has up to 20GB of storage)._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Video_Download_v1_00.tsv.gz to ../../../../../../tmp/recsys/amazon_reviews_us_Digital_Video_Download_v1_00.tsv.gz\n"
     ]
    }
   ],
   "source": [
    "!mkdir /tmp/recsys/\n",
    "!aws s3 cp s3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Video_Download_v1_00.tsv.gz /tmp/recsys/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the data into a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) so that we can begin to understand it.\n",
    "\n",
    "*Note, we'll set `error_bad_lines=False` when reading the file in as there appear to be a very small number of records which would create a problem otherwise.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Skipping line 92523: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 343254: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 524626: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 623024: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 977412: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 1496867: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 1711638: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 1787213: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 2395306: expected 15 fields, saw 22\n",
      "\n",
      "Skipping line 2527690: expected 15 fields, saw 22\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>marketplace</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>review_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_parent</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_category</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>helpful_votes</th>\n",
       "      <th>total_votes</th>\n",
       "      <th>vine</th>\n",
       "      <th>verified_purchase</th>\n",
       "      <th>review_headline</th>\n",
       "      <th>review_body</th>\n",
       "      <th>review_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>US</td>\n",
       "      <td>12190288</td>\n",
       "      <td>R3FU16928EP5TC</td>\n",
       "      <td>B00AYB1482</td>\n",
       "      <td>668895143</td>\n",
       "      <td>Enlightened: Season 1</td>\n",
       "      <td>Digital_Video_Download</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>I loved it and I wish there was a season 3</td>\n",
       "      <td>I loved it and I wish there was a season 3... ...</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US</td>\n",
       "      <td>30549954</td>\n",
       "      <td>R1IZHHS1MH3AQ4</td>\n",
       "      <td>B00KQD28OM</td>\n",
       "      <td>246219280</td>\n",
       "      <td>Vicious</td>\n",
       "      <td>Digital_Video_Download</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>As always it seems that the best shows come fr...</td>\n",
       "      <td>As always it seems that the best shows come fr...</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US</td>\n",
       "      <td>52895410</td>\n",
       "      <td>R52R85WC6TIAH</td>\n",
       "      <td>B01489L5LQ</td>\n",
       "      <td>534732318</td>\n",
       "      <td>After Words</td>\n",
       "      <td>Digital_Video_Download</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Charming movie</td>\n",
       "      <td>This movie isn't perfect, but it gets a lot of...</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>US</td>\n",
       "      <td>27072354</td>\n",
       "      <td>R7HOOYTVIB0DS</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>239012694</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>Digital_Video_Download</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Five Stars</td>\n",
       "      <td>excellant this is what tv should be</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>US</td>\n",
       "      <td>26939022</td>\n",
       "      <td>R1XQ2N5CDOZGNX</td>\n",
       "      <td>B0094LZMT0</td>\n",
       "      <td>535858974</td>\n",
       "      <td>On The Waterfront</td>\n",
       "      <td>Digital_Video_Download</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>Y</td>\n",
       "      <td>Brilliant film from beginning to end</td>\n",
       "      <td>Brilliant film from beginning to end. All of t...</td>\n",
       "      <td>2015-08-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  marketplace  customer_id       review_id  product_id  product_parent  \\\n",
       "0          US     12190288  R3FU16928EP5TC  B00AYB1482       668895143   \n",
       "1          US     30549954  R1IZHHS1MH3AQ4  B00KQD28OM       246219280   \n",
       "2          US     52895410   R52R85WC6TIAH  B01489L5LQ       534732318   \n",
       "3          US     27072354   R7HOOYTVIB0DS  B008LOVIIK       239012694   \n",
       "4          US     26939022  R1XQ2N5CDOZGNX  B0094LZMT0       535858974   \n",
       "\n",
       "                           product_title        product_category  star_rating  \\\n",
       "0                  Enlightened: Season 1  Digital_Video_Download            5   \n",
       "1                                Vicious  Digital_Video_Download            5   \n",
       "2                            After Words  Digital_Video_Download            4   \n",
       "3  Masterpiece: Inspector Lewis Season 5  Digital_Video_Download            5   \n",
       "4                      On The Waterfront  Digital_Video_Download            5   \n",
       "\n",
       "   helpful_votes  total_votes vine verified_purchase  \\\n",
       "0              0            0    N                 Y   \n",
       "1              0            0    N                 Y   \n",
       "2             17           18    N                 Y   \n",
       "3              0            0    N                 Y   \n",
       "4              0            0    N                 Y   \n",
       "\n",
       "                                     review_headline  \\\n",
       "0         I loved it and I wish there was a season 3   \n",
       "1  As always it seems that the best shows come fr...   \n",
       "2                                     Charming movie   \n",
       "3                                         Five Stars   \n",
       "4               Brilliant film from beginning to end   \n",
       "\n",
       "                                         review_body review_date  \n",
       "0  I loved it and I wish there was a season 3... ...  2015-08-31  \n",
       "1  As always it seems that the best shows come fr...  2015-08-31  \n",
       "2  This movie isn't perfect, but it gets a lot of...  2015-08-31  \n",
       "3                excellant this is what tv should be  2015-08-31  \n",
       "4  Brilliant film from beginning to end. All of t...  2015-08-31  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews = pd.read_csv('/tmp/recsys/amazon_reviews_us_Digital_Video_Download_v1_00.tsv.gz', delimiter='\\t',error_bad_lines=False)\n",
    "reviews.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see this dataset includes information like:\n",
    "\n",
    "- `marketplace`: 2-letter country code (in this case all \"US\").\n",
    "- `customer_id`: Random identifier that can be used to aggregate reviews written by a single author.\n",
    "- `review_id`: A unique ID for the review.\n",
    "- `product_id`: The Amazon Standard Identification Number (ASIN).  `http://www.amazon.com/dp/<ASIN>` links to the product's detail page.\n",
    "- `product_parent`: The parent of that ASIN.  Multiple ASINs (color or format variations of the same product) can roll up into a single parent parent.\n",
    "- `product_title`: Title description of the product.\n",
    "- `product_category`: Broad product category that can be used to group reviews (in this case digital videos).\n",
    "- `star_rating`: The review's rating (1 to 5 stars).\n",
    "- `helpful_votes`: Number of helpful votes for the review.\n",
    "- `total_votes`: Number of total votes the review received.\n",
    "- `vine`: Was the review written as part of the [Vine](https://www.amazon.com/gp/vine/help) program?\n",
    "- `verified_purchase`: Was the review from a verified purchase?\n",
    "- `review_headline`: The title of the review itself.\n",
    "- `review_body`: The text of the review.\n",
    "- `review_date`: The date the review was written.\n",
    "\n",
    "For this example, let's limit ourselves to `customer_id`, `product_id`, and `star_rating`.  Including additional features in our recommendation system could be beneficial, but would require substantial processing (particularly the text data) which would take us beyond the scope of this notebook.\n",
    "\n",
    "*Note: we'll keep `product_title` on the dataset to help verify our recommendations later in the notebook, but it will not be used in algorithm training.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = reviews[['customer_id', 'product_id', 'star_rating', 'product_title']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because most people haven't seen most videos, and people rate fewer videos than we actually watch, we'd expect our data to be sparse.  Our algorithm should work well with this sparse problem in general, but we may still want to clean out some of the long tail.  Let's look at some basic percentiles to confirm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "customers = reviews['customer_id'].value_counts()\n",
    "products = reviews['product_id'].value_counts()\n",
    "\n",
    "quantiles = [0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.96, 0.97, 0.98, 0.99, 1]\n",
    "product_q = pd.DataFrame(zip(quantiles, products.quantile(quantiles)), columns=[\"quantile\", \"products\"])\n",
    "customer_q = pd.DataFrame(zip(quantiles, customers.quantile(quantiles)), columns=[\"quantile\", \"customers\"])\n",
    "# product_q.tail(10)\n",
    "# customer_q.tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAELCAYAAAD+9XA2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi41LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvSM8oowAAGvFJREFUeJzt3X+UXGWd5/F3dSckEpNuo2GAHddRhnw5gDqLhhE3IawKAzmsjj8YwR+c6CA4MophTnQQR8KyzLByFsZVHBcVMvHHIoqsGhd0IouEg6thWVFRv0EZPHP4MQRJx44JCZ2u/ePehrJJp6urn0qnyft1Tk7qPvU8Tz33dnd96rn31r2NZrOJJEmT1TPVA5AkPTMYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkImZM9QD2Iq8xI0kT12i34v4UKGzaNDjVQ5CkaWPBgrkTqu8uL0lSEQaKJKkIA0WSVISBIkkqwkCRJBVhoEiSijBQJElFGCiSpCL2qy82StL+Yt7MXTR2DXXUttk7g9880TvhdgaKJD0DNXYN8cjKMztqe9Dla4CJB4q7vCRJRRgokqQiDBRJUhEGiiSpCANFklSEgSJJKsJAkSQVYaBIkoowUCRJRRgokqQiDBRJUhEGiiSpCANFklRE1642HBG/B9wIPEF12cp3A78EVgOHAj8Bzs3M4YhYBHwMaACXZubauo9VwGuAQeDtmfloRDwP+BwwF1iXmau6tQ6SpPZ1c4byKLA4M5cCHwb+GngncGdmLgGGgZPrulcCpwEnApdERG9EHAUsyszFwDXAB+q6HwSuqcsXRcSRXVwHSVKbuhYombkrM4frxX7gbmAJsLYuWwscHxGzgRmZ+UBmbgU2Aofvpu6S+vHi0X10ax0kSe3r6g226tnDZ4DnA2+kmoFsrp8eAObX/wZamrWW3weQmdsj4tn183Myc3tL3Re2O57+/gM7WxFJmmYaW3d23Lanp0H/vIm/X3Y1UDLzp8ArI+KlwNXA/VSzlYeBPuAxqoDpb2n2tPJ6FrO1fn5bRMzOzMdb6rZlYGDbZFZHkqaNvp5mx22Hh5tsGdjGggVzJ9Sua7u8ImJWy+IAsA24DVhWly0D1tezjaGIOCQi5lDt7vpFXfeUlrq314/Xt/RxSr0sSZpi3ZyhHBMRl1EdfG8A5wM/B1ZHxG3Az4Cb6rp/BdxQ17s4M4eAeyLi7oi4neosr5GbI38UWBMR5wO3ZOY9XVwHSVKbGs1m59Oiaaa5adPgVI9BkvaKvp4dPLLyzPEr7sZBl69hy/CskV1ejXbb+cVGSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkImZ0q+OIOA64AtgJbAXeCrwfOA3YBDycmafXdU8FLgSawHmZuSEieoCrgKOBB4Hlmbk9Ig4DPgscAFybmZ/u1jpIktrXzRnKr4BXZ+ZS4BvAuXX5RZl5QkuY9AKXACdRhc2Vdb1TgOHMXALcCbyjLr8MWAksAc6KiPldXAdJUpu6FiiZ+WBmbqsXdwJD9eMPR8T6iDijXj4c2JiZg5n5ADAzImZTBcbaus5a4Pj68cLM3JCZu4BbgWO7tQ6SpPZ1bZfXiIh4LvAe4GSgmZmrIqIPuCUibgfmA5tbmgzUZa3lI2UAjd3UbUt//4EdrYMkTTeNrTs7btvT06B/3sTfL7saKBFxIPBl4H2Z+ehIeWZuiYjvUB0fuR/ob2nWBzxGFSb9o8qgOs4yum5bBga2jV9Jkp4B+nqa41caw/Bwky0D21iwYO6E2nVtl1dEzACuAz6emXfUZX0tzx0H/BK4F1gYEXMi4mBgKDMfB24DltXdLQPW1483RsQx9bGXpcAPurUOkqT2dXOGcgbVcY95EXEe8E0gIuJIoBf4YmZupCpcBayjmn2sqNvfBLw2ItYDDwHL6/ILqM7ymgmszsy2ZyiSpO5pNJudT4ummeamTYNTPQZJ2iv6enbwyMozO2p70OVr2DI8a2SXV2O8+iP8YqMkqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRM7rVcUQcB1wB7AS2Am+tX+9zwFxgXWauquueClwINIHzMnNDRPQAVwFHAw8CyzNze0QcBnwWOAC4NjM/3a11kCS1r5szlF8Br87MpcA3gHOBDwLXZOZiYFFEHBkRvcAlwEnAacCVdftTgOHMXALcCbyjLr8MWAksAc6KiPldXAdJUpvamqFExOnANzNzMCI+AvwRcFlm/mCsNpn5YMviTmAIWAx8pC5bCxwPDAMbM3MQGIyImRExmyow1rbUvQj4JLAwMzfU47oVOBa4uZ316O8/sJ1qkjTtNbbu7LhtT0+D/nkTf79sd5fXBZl5XUS8Cngl8LfAJ6jezPcoIp4LvAc4GXhrZm6vnxoAXgjMBza3NBmoy1rLR8oAGrup25aBgW3tVpWkaa2vp9lx2+HhJlsGtrFgwdwJtWt3l9dIvdcBn83M24CZ4zWKiAOBLwPvy8xHgW317AOgD3iMKjT6W5rtrnykDKrjLKPrSpKmWLuB8sN699KpwP+KiLn87hv700TEDOA64OOZeUddvB5YVj8+pV6+F1gYEXMi4mBgKDMfB25rqbusrguwMSKOqY+9LAXG3O0mSdp72t3ldS7wh8B9mfnbejfWe8dpcwbVMZJ5EXEe8E3go8CaiDgfuCUz7wGIiFXAOqqQWlG3vwl4bUSsBx4CltflF1Cd5TUTWJ2ZzlAkaR/QaDbH388WEXdl5jHjle3jmps2DU71GCRpr+jr2cEjK8/sqO1Bl69hy/CskWMojfHqj9jjDCUiXgQcDsyNiJNanpoHzOpkoJKkZ6bxdnm9GPhTqjOpTueppBoE3tXFcUmSppk9Bkpmfg34WkQcu6fvnEiS1O5ZXhdFxJOn9kbEcyLi610akyRpGmo3UA7NzIGRhczcDDy/O0OSJE1H7QbKzohYOLIQEQE80Z0hSZKmo3a/h7IC+GZE3Et1YP5FQGfno0mSnpHaCpTMvCMijgKiLvp5ZjpDkSQ9qd2rDZ89qui4iCAzr+7CmCRJ01C7u7wOaXk8i+reJT8HDBRJEtD+Lq+LW5cj4mLavAeJJGn/0OkdG5+Lpw1Lklq0ewzlIaorATfq/7fy1J0XJUlqe5fXIePXkiTtz8a72vDxe3q+vnOjJEnjzlD+ov7/ucArqO6iCLAE+F7LsiRpPzfe1YbPAIiIbwFHZOaD9fKhVHdNlCQJaP8srxcAD7csPwK8sPxwJEnTVbtfbPwycHtEfJXqLK/XA9d3bVSSpGmnrRlKZv4NcB7VFYaHgBWZ6WnDkqQntTtDAfhX4MGWx5IkPamtGUpEvAv4FvDHVGd73RQRf97NgUmSppd2ZyjvBRZl5laAiLgIuAPP9JIk1doNlEb9b3TZmCJiJnArcBRwVmZ+JSJWAacBm4CHM/P0uu6pwIVUB/zPy8wNEdEDXAUcTbWrbXlmbo+Iw6iC7ADg2sz8dJvrIEnqonYD5Qrgrvr7KA3gVcB/GafNEPAm4JxR5Rdl5ldGFiKiF7gEOB6YB3wJWAycAgxn5pKIWAm8A/gkcBmwErgLuCMibsjMx9pcD0lSl4x7DCUiGsA64ATgn4BvA6/OzNV7apeZzcx8aDdPfTgi1kfEGfXy4cDGzBzMzAeAmRExm+rb+GvrOmupAgdgYWZuyMxdVDOgY8dbB0lS9407Q8nMZkR8MzNfAjwwydf7eGauiog+4JaIuB2YD2xuqTNQl7WWj5TB7+5qay0fV3//gZ2OW5KmlcbWnR237elp0D9v4u+X7e7yuj0iXpOZ6yb8Ci0y89f1/1si4jtUx0fuB/pbqvUBj1GFSf+oMqiOs4yu25aBgW0djVuSppu+nub4lcYwPNxky8A2FiyYO6F27QbKG4BzImIQePJdOTMPnciLRURfHSYzgOOAzwD3AQsjYg4wFxjKzMcj4jZgGdWdIZcB6+tuNkbEMcDdwFLgoxMZgySpOyZyT/k3UB0sb1K9uf/P8RpFxPXAy4GtEXEsMD8ijgR6gS9m5sa63iqq4zRNYEXd/CbgtRGxHngIWF6XX0B1ltdMYLUH5CVp39BoNsefFkXEp4DnAdfVRX8G/Doz/2LsVvuc5qZNg1M9BknaK/p6dvDIyjM7anvQ5WvYMjxrZJfXHr8i0qrdGcrxmXnkyEJE3AD8dGJDlCQ9k7V7+fqfRsSLW5aPAn7ShfFIkqapdmcoL6L6YmPWy0cAP4qI7wHNzHxlV0YnSZo22g2U13d1FJKkaa+tQMnMX3V7IJKk6a3dYyiSJO2RgSJJKsJAkSQVYaBIkoowUCRJRRgokqQiDBRJUhEGiiSpCANFklSEgSJJKsJAkSQVYaBIkoowUCRJRRgokqQiDBRJUhEGiiSpCANFklSEgSJJKsJAkSQV0dY95TsRETOBW4GjgLMy8ysR8Tzgc8BcYF1mrqrrngpcCDSB8zJzQ0T0AFcBRwMPAsszc3tEHAZ8FjgAuDYzP92tdZAkta+bM5Qh4E3A37eUfRC4JjMXA4si4siI6AUuAU4CTgOurOueAgxn5hLgTuAddfllwEpgCXBWRMzv4jpIktrUtRlKZjaBhyKitXgx8JH68VrgeGAY2JiZg8BgRMyMiNlUgbG2pe5FwCeBhZm5ASAibgWOBW5uZ0z9/QdOZpUkadpobN3Zcduengb98yb+ftm1QBnDnMzcXj8eAF4IzAc2t9QZqMtay0fKABq7qduWgYFtHQxZkqafvp5mx22Hh5tsGdjGggVzJ9Rubx+U31bPPgD6gMeoQqO/pc7uykfKoDrOMrquJGmK7e1AWQ8sqx+fUi/fCyyMiDkRcTAwlJmPA7e11F1W1wXYGBHH1MdelgI/2GujlySNqau7vCLieuDlwNaIOBb4KLAmIs4HbsnMe+p6q4B1VLOPFXXzm4DXRsR64CFgeV1+AdVZXjOB1ZnpDEWS9gGNZrPz/WzTTHPTpsGpHoMk7RV9PTt4ZOWZHbU96PI1bBmeNXIMpTFe/RF+sVGSVISBIkkqwkCRJBVhoEiSijBQJElFGCiSpCIMFElSEQaKJKkIA0WSVISBIkkqwkCRJBVhoEiSijBQJElFGCiSpCIMFElSEQaKJKkIA0WSVISBIkkqwkCRJBVhoEiSijBQJElFGCiSpCIMFElSETOm4kUj4rfAhnrxMuC7wGrgUOAnwLmZORwRi4CPAQ3g0sxcW7dfBbwGGATenpmP7tUVkCQ9zVTNUP45M0+o/90MvBO4MzOXAMPAyXW9K4HTgBOBSyKiNyKOAhZl5mLgGuADUzB+SdIoUzJDAZ4fEbcB/wK8D1gCXFw/txZYGhG3ADMy8wGAiNgIHF7XXdtS9/y9OXBJ0u5NVaAclpmPRsQ7gUuB+cDm+rmBenl+/ZjdlN8HkJnbI+LZ7b5of/+BBYYuSfu+xtadHbft6WnQP2/i75dTEigtxzyuA84B7gf6gYeBPuAxqoDpb2n2tPKImA1sbfd1Bwa2TXLkkjQ99PU0O247PNxky8A2FiyYO6F2e/0YSkTMiYjeenEpcC9wG7CsLlsGrM/M7cBQRBwSEXOodnf9oq57Skvd2/fa4CVJY5qKGcoRwGci4jfADuAs4FFgdX1c5WfATXXdvwJuoDrL6+LMHALuiYi7I+J2qrO8ztzbKyBJerpGs9n5tGiaaW7aNDjVY5CkvaKvZwePrOzs8/ZBl69hy/CskV1ejXbb+cVGSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkIgwUSVIRBookqQgDRZJUhIEiSSrCQJEkFWGgSJKKMFAkSUUYKJKkImZM9QA6ERHvAt4B7ATemZn3TfGQJGlSep81k6Fm5+1nNGDX9ifKDaiTMUzpq3cgIuYDZwH/Hvh3wGXAn03poCTtd2bP7qHZ3NVx+0ajl8cfH35yeagJb/vSXR339/k3H0Oj49ZlTLtAAY4Fbs3MIWBDRMREGk/mU8DoTwDzZu6isWuos86AZu8MfvNE75PLcw/cBc3O+6Mxg8FtT/V3wOxhdg131l9vzwx2Pv7UHtHSfzylP42V/FnsSz8H2Ld/Fvvz30SzuYurr76646GdffbZMOURUFaj2ZzEX/UUiIi3AP82My+rl3+cmS9uo+n0WlFJ2je0nXrTcYayGXhJy3K7H9WeWR8FJGkfMx0D5fvARyKiF3gpcO8Uj0eSxDQMlMx8LCL+EVgPPAH8+RQPSZLENDyGIknaN/nFRklSEQaKJKkIA0WSVISBIkkqYtqd5VXKnq4HFhGLgI9RfXfl0sxcW6KPiDgU+AYQwAmZeeck+3sd8CFgB/AAcGZmPtFhXy8F/qGuPwy8LTMfnOz2ioi3Af+QmXMnua5/APxf4Md1tfdn5g8n+DP5BPBG4IbM/MvRbcfrIyL6gRuAA+qxvS8z74qIE4Av8NQp7Kdn5sMT6bt+7lZgJtXZizePfHl3gn18ux4fwHHAwUAfk9925wJvq59bkZm7vUbIRPsYb9t1MqaIOAs4ner97dLM/KcOx/YW4Oy6yu8DX8/M8yexrv3A9VQ/n4n+je2uv4OBf6z7+zFwXmY+7SyriJgJ3AocBZyVmV8Z9fypwIVUX/4+LzM3jO6jXftloLRxPbArgdOALcD6iLgpM3dNtg/gMeAk4L+WGBPwQ2BxZj4RESP1v9BhXz/NzFfWY1kOvAf48GS2V/2LfBrwLwXWFeB7mXnq6G03gX7/lioQ3thhHzuoQvuBiDgC+DhwYv3cjXsKqTbHB/C6zHy00z4y86S63lHAlZm5OSL6mMS2i4jfA94KLAaeB1wH/IeCfex223XSX0S8GFiUma+ZbF+Z+UXgi3WdL1P97kxmXU8D1mfmJRHxduDdwEcm0d+HgKsy8+sRcQXwJ8DNo7cjMAS8CThnN9u4F7gEOB6YB3ypfp2O7K+7vJ68Hlidxk9eDywiZgMzMvOBzNwKbAQOL9FHZj6emb8uNabM/NXIjITqk8vQJPpqvUxpH/CjAtvrHGA11aexSa1r/fSiiFgfEZ+MiGdNZBsC1J8GxztPfsw+MnN7Zj5QL7Zub4BTI+L2iLg0Isa6KsMex1eP7caI+FZE/FGHfYx4C/A/WpYns+3+ALgnM3dl5r8Cvx8Rswr2Mda266S/NwC7ImJdRHwxIp4z2fWLiLnA0cAdkxzbz4CRmXo/sGmS/R1ONfMEuAtYym5kZjMzH9rdc3UfGzNzsP7dnln/DXZkfw2U+VSXcBnRM+q5gZblgbqsG30U6S8iDqP6dHLjZPqKiFdFxA+oZid3jWozof4iYg5wcmbeyNN1Mr6HgD/MzCXAo8B5E+y3XeP2Ub/pXQF8tC66k+oN4HjgUODNHfZ9Wr1+5wPXdjq+2uuBr9aPJ7vtfgG8LCIOjIiFwAuA5/B0nfSxp23XSX+HAnPqGcotwF8XWL83AF8btTupk/5+AiyOiB8Dfwl8fpL9/ZhqjwdUM+Xd/UzGM/p123mvGtP+GiibqT4hjNi1h+f6qHZVdaOPSfcXEQdR/WK+LTN3TqavzLwlM48FVlLtHprM2FYAnxi9kp32l5k7MnOwLruOarfARPptVzt9fIzq0+T/BsjMrfX4hoEvjzG2cfse2dWVmfcAT4wxkxh3fBHxiqqb3FL3N6ltV8+qLwNuotoN+iOqYJp0H+Nsu07GtBn4dl3tZmDkwrGTWb8z+N3ZXqf9fQD4fFYXs13BUx9IOu3v74ATI2Id8FuqDw4T1cl71Zj210D5PrA0Inoj4hhargeWmduBoYg4pP6UfTjVJ4Ru9DGp/iLi2VT7dVdk5r2T7Kt1F8YAsH2S63oE8P6IuBl4QURcM8nxzWtpv5TdX8NtzH4nYI99RMSHgKHM/PuWsr42xtZO3/Pq/w+m+pS9faJ91N5Cve+/td9xxrfHfjPz+sxcClwM/Dyr20dMuo9xtl0nY/ou8LK6ysuBX05m/eoPbIdk5t0FtlcPTwXVr/ndGcWE+8vMzZn5lno21stTeygm4l5gYUTMqX/vhjLz8Q76AfbjS69ExLuBM3nqemCLgX/OzO9GxB9THRhuAH+XmV8v0UdUB6lvAo6kOlD9pcy8YhL9/Q1wLvDzuovPZubnOuzrDVS7QobrNmdn5v0ltldE/CQzj57ktjuV6o9pK9WnquWZ2bprrJ1+L6TafXEQ1UkIfzK6/Z76AO4D7gdupzre8UBmvjUizgHeRRXCvwTelb97TKqdvtcDG4BtVCfLXJiZt0ykj3ode6l+H14yEkiFtt0XgEPq9ufm2GexTaiP8bZdB/31AP+N6pjHENVJFA92un5RnV31nMz8zwXW9d8Aa6je/A8A3pMtZ9t10N+JVAfmm1TvJf99dz+Tuu/rqQJ2K9XM7TGqkyEyIl4LXFD3syIzvz9WP+PZbwNFklTW/rrLS5JUmIEiSSrCQJEkFWGgSJKKMFAkSUUYKNI+KCKWR8TzWpbX1f+fEBHXTd3IpLEZKNK+aTnVhQAByFEXO5T2Rfvl1YalEiLiEqorwv6K+lL7wHWZeXD9/Crg4cz8VERcDCwDnkV1GfQP1XXup7oE+Z9SXV35VODVVF9CuzEiHs3MJRHx8Ei/La//bKpbDhxRF703M/9P99ZY2jNnKFIHIuJYqgvyHU11D4tXjNPkY5m5CHgJ8JKo7j8z4heZ+VLgbuDN9QU17wReX1/QcSwfprq3yyKqi0Fe1dnaSGUYKFJnXgl8NTOfqC/7vX6c+q+OiA1U97BZRHX5nREjl6r5f1SXKm/XicDFEfFDYC1wUES410FTxl8+qXPN3TxuLZsFT97j5QrgZZn5SFR3jmy9GOeO+v9hqus8tasBnJItd/2TppIzFKkzdwCvj4iZUd3aeWTX1G8jYuQGSCP3qphNFRabo7oz339so/9BnroZ01jWUV0cFIBRu9Gkvc5AkTqQmT8AvkN106RrgZGD4RdRXUL920DWdQeo7qfxM6qbXn2vjZdYDXwuIva0K+0/AYdGxI8i4qdUt5CVpoxXG5YKqL8b8qnMvHWqxyJNFWcokqQinKFIkopwhiJJKsJAkSQVYaBIkoowUCRJRRgokqQi/j9TCUwwtlyb2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "axp = sns.barplot(x=\"quantile\", y=\"products\", data=product_q, palette=five_thirty_eight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAELCAYAAADOeWEXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi41LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvSM8oowAAGB9JREFUeJzt3X+U3XV95/HnTBKJxGSmqWGBtT+owpuDqFUaKiwBVhaUbAqioiCUDRZqF1oRethugeVHKdaDZ0EWoT9WUbBSRNRdTAUtskA4tBZKbfHXG9DC9vBjCZKJCQmEZO7+8f0OuYyTme/n3rmZm+T5OCdn5n7n+33f9/3eyX3N99fnO9BqtZAkqanBmW5AkrR9MTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRWbPdAM94BgqktSZgSYz7YjBwapVa2e6BUnarixaNL/xvO6qkiQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUZIe8AFCSdhYL5mxmYPOmjpdvzZrNT1+aVbSMwSFJ27GBzZt45txTOl5+t0/cAJQFh7uqJElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklSkZ2NVRcRBwBXARmAdcBLwUeB4YBXwdGaeUM+7DDgfaAFnZeb9ETEIXAPsDzwJLM/MDb3qV5LUTC+3OB4HjsjMw4CvAWfW0y/KzMPbQmMWcClwFFWoXFnPdzQwmplLgAeAU3vYqySpoZ4FR2Y+mZnr64cbgbFxfy+IiJURcWL9eG/g4cxcm5lPAHMiYi6wBFhRz7MCOLRXvUqSmuv5sOoR8fPAGcC7gFZmXhwRQ8CdEXEvsBBY3bbISD2tffrYtEaGh3edjtYlqe8NrNvY1fKDgwMMLyj7zOxpcETErsCXgI9k5rNj0zNzTUR8i+r4xWPAcNtiQ8BzVKExPG5aIyMj66eeSZJ2AEODra6WHx1tsWZkPYsWzW+8TM92VUXEbOAm4OrMvK+eNtT2s4OAHwGPAPtExLyI2B3YlJkvAPcAS+tyS4GVvepVktRcL7c4TqQ6LrEgIs4C/hqIiNiP6nZTN2bmw1QTLwbuoDqr6ux6+duAYyJiJfAUsLyHvUqSGhpotbrbzOlDrVWr1s50D5K0TQwNvtj1rWPXjO4ytqtqoMkyXgAoSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIrN7VTgiDgKuADYC64CT6uf7PDAfuCMzL67nXQacD7SAszLz/ogYBK4B9geeBJZn5oZe9StJaqaXWxyPA0dk5mHA14AzgT8ArsvMQ4DFEbFfRMwCLgWOAo4HrqyXPxoYzcwlwAPAqT3sVZLUUM+2ODLzybaHG4FNwCHAhfW0FcChwCjwcGauBdZGxJyImAssqecZm/ci4Nomzz08vGv3L0CStgMD6zZ2tfzg4ADDC8o+M3sWHGMi4ueBM4B3ASe17W4aAfYCFgKr2xYZqae1Tx+b1sjIyPouu5ak7cPQYKur5UdHW6wZWc+iRfMbL9PTg+MRsSvwJeAjmfkssL7emgAYAp6jCofhtsUmmj42TZI0w3oWHBExG7gJuDoz76snrwSW1t8fXT9+BNgnIuZFxO7Apsx8Abinbd6l9bySpBnWy11VJ1Idw1gQEWcBfw1cDtwQEecAd2bm9wAi4mLgDqqzqs6ul78NOCYiVgJPAct72KskqaGBVqu7/WN9qLVq1dqZ7kGStomhwRd55txTOl5+t0/cwJrRXcaOcQw0WcYLACVJRQwOSVIRg0OSVMTgkCQVMTgkSUUMDklSEYNDklTE4JAkFTE4JElFDA5JUhGDQ5JUxOCQJBUxOCRJRQwOSVIRg0OSVMTgkCQVMTgkSUUaBUdEnBAR8+vvL4yIr0TEgb1tTZLUj5pucfxhZq6NiHcABwOfBD7Vu7YkSf2qaXDMqr8eC3wmM+8B5vSmJUlSP2saHP8YEXcBy4Cv17utWj3rSpLUt6YMjogYBP4COAc4IDOfB14FnNrj3iRJfWj2VDNk5mhEXJWZb2ub9hPgJz3tTJLUl6YMjtrtEXEacAuwfmxiZm7sSVeSpL7VNDhOrL+eT3VsY6D++iu9aEqS1L8aBUdm7tXrRiRJ24dGwRERi4ALgT0z870RsS9wYGbe0NPuJEl9p+npuNcDdwNvqB8/Cpzbk44kSX2taXAsysxbgFGAzNwEbO5ZV5KkvtU0OFZHxC9QX/QXEUcCz/SsK0lS32p6VtUZwKeBfSLih8Bq4OTJFoiIOcBdwBuB0zLzloi4GDgeWAU8nZkn1PMuY8sZW2dl5v31hYfXAPsDTwLLM3ND2cuTJE23pmdVPQq8MyJeAwxm5k8bLLYJeB/w4XHTL6p3ewEQEbOAS4FDgQXAF4FDgKOB0cxcEhHnUl2pfm2TfiVJvdP0rKoFwAeAXwRmRQQAmXne1pbJzBbw1Ni8bS6IiLOAazPzr4C9gYczcy2wNiLmRMRcYAmwol5mBXARBockzbimu6puA/4O+C71AfIOXZ2ZF0fEEHBnRNwLLKTa9TVmpJ7WPn1sWiPDw7t20aIkbT8G1nU3gMfg4ADDC8o+M5sGxy6Z+fvlLb1SPcYVmbkmIr5FdfziMWC4bbYh4Dmq0BgeN62RkZH1U88kSTuAocHuBiofHW2xZmQ9ixbNb7xM0+C4NiLOAb4JvBxvmflwSYMRMVSHxmzgIKoD7j+mOug+D5gPbMrMFyLiHmApcHv9dWXJc0mSeqNpcOwFLKe6kdPYrqoW8I7JFoqIm4FfA9bVt5pdGBH7Ud0Y6sax4KnPtrqjrnl2vfhtwDERsRJ4qn5+SdIMaxocJwCvLx0NNzPf33C+W4Fbx00b5WfPyJIkzbCmFwA+AOzRy0YkSduHplscAfwgIr5PdYxjAGhl5sE960yS1JeaBsdxPe1CkrTdaLSrKjMfpzpwfWD9r1VPkyTtZBoFR0ScDnwD+HXg7VS3kv2tXjYmSepPTXdV/R6wODPXAUTERcB9wGd61ZgkqT81PatqoP43fpokaSfTdIvjCuDBiPgGVWC8A7i8Z11JkvpW04PjnwUOB/6GatiRI+ppkqSdTNNh1e/OzMOAJyaYJknaiUwaHBGxEFgEvDYi9mbLcY0FwG497k2S1Iem2uL4DarBBX8R+HO2BMdaqlu9SpJ2MpMGR2ZeD1wfEe/OzP+1jXqSJPWxpmdVvT4i5gPPU91D41eBCzLz6z3rTJLUl5pex3FyfU/w/wi8Gng38Mc960qS1LeaBsfc+uuxwBcy8/8WLCtJ2oE03VW1IiJ+THVQ/D9HxCLabiErSdp5NL0A8FyqW8C+LTNfojrWcUwvG5Mk9aemFwB+rO379h+dN90NSZL6W9NdVdn2/S7AMtquIpck7TwaBUd9PcfLIuLTwL096UiS1Nea7qp6VdvDQeCtVEORSJJ2MiW7qlpUQ45sBh4DTu9RT5KkPtb0WozrgQMyc6/MfANwPHBo79qSJPWrpsFxbGauHntQf39cb1qSJPWzpsExKyLmjT2ox61quptLkrQDafrhfzWwMiJuqh+fAFzZm5YkSf2s6ZXj/xP4T8CG+t9vZuZ1vWxMktSfGu9uysyHgId62IskaTvgCLeSpCI9O8AdEXOAu4A3Aqdl5i0R8Vrg88B84I7MvLiedxnVrWhbwFmZeX9EDALXAPsDTwLLM3NDr/qVJDXTyy2OTcD7gE+2TfsD4LrMPARYHBH7RcQs4FLgKKrrQ8YOuh8NjGbmEuAB4NQe9ipJaqhnwZGZrcx8atzkQ4AV9fcrqC4i3Bt4ODPXZuYTwJyImAssmWBeSdIM29bXYsxr2900AuwFLARWt80zUk9rnz42rZHh4V2771SStgMD67q7p97g4ADDC8o+M7d1cKyPiLmZ+QIwBDxHFQ7DbfNMNH1sWiMjI+unp1tJ6nNDg62ulh8dbbFmZD2LFs1vvMy2PqtqJbC0/v7o+vEjwD4RMS8idgc21cFyT9u8S+t5JUkzrKdbHBFxM9UtZ9dFxIHA5cANEXEOcGdmfq+e72LgDqqzqs6uF78NOCYiVgJPAct72askqZmBVqu7zZw+1Fq1au1M9yBJ28TQ4Is8c+4pHS+/2yduYM3oLmO7qgaaLOMFgJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSoye1s/YUQ8D9xfP/w4cDfwOWBP4LvAmZk5GhGLgauAAeCyzFyxrXuVJP2smdji+JfMPLz+dzvwIeCBzFwCjALvque7EjgeOBK4NCJmzUCvkqRxtvkWB/ALEXEP8K/AR4AlwCX1z1YAh0XEncDszHwCICIeBvYGftjkCYaHd532piWpHw2s29jV8oODAwwvKPvMnIngeH1mPhsRHwIuAxYCq+ufjdSPF9bfM256IyMj66epVUnqb0ODra6WHx1tsWZkPYsWzW+8zDbfVZWZz9bf3gS8lSo0hutpQ8Bz46a1T5ckzbBtGhwRMa/tWMVhwCPAPcDSetpSYGVmbgA2RcQeETGPajfVo9uyV0nSxLb1rqp9gU9HxE+BF4HTgGeBz9XHPX4A3FbP+/vAl6nOqrokMzdt414lSRPYpsGRmf9AtXtqvPdPMO+3gYN73pQkqYgXAEqSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIgaHJKmIwSFJKmJwSJKKGBySpCIGhySpiMEhSSpicEiSihgckqQiBockqYjBIUkqYnBIkooYHJKkIrNnuoGpRMTpwKnARuBDmfnjGW5JknZqfb3FERELgdOAQ4FzgY/PbEeSpH7f4jgQuCszNwH3R0Q0XXDWq+ewqdX5E88egM0bXnr58YI5mxnYvKmjWq1Zs/npS7Nefjx/183Q6qwWAAOzWbt+S71XzR1l82jn9WYNzmbjC1v+hpg7d5BWa3NnrQ3M4oUXRrfU7qP3Aab5vejj9wF6+1701fsAff1e9Pr/xEzo9+BYCKxue9xoC2nRovnT8+yvmTs9dYBF01apMnfeNBecplUGMH8aawHT+j7A9L4X/fw+QH+/F/6f6MIr3of5vO66r3dVrvS96PfgWA28ue1xk8gf6FEvkiT6Pzi+DVwYEbOAtwCPzHA/krTT6+vgyMznIuJ6YCXwEvBbM9ySJO30BlqtLo7SSJJ2On19Oq4kqf8YHJKkIgaHJKmIwSFJKtLXZ1VNh8nGuoqIxcBVVNd+XJaZK6ajRkTsCXwNCODwzHygy3rHAucBLwJPAKdk5ktd1HsL8Kf1/KPAyZn5ZDfrKyJOBv40M19xqVMHvf0y8A/AQ/VsH83M74x/TxrU/hTwXuDLmfm7Ey0/WY2IGAa+DLyq7u8jmflgRBwOfIEtp4afkJlPF/Z2FzCH6kzB2zNzwqF0pqjxzbo3gIOA3YEhGqy7KeqeCZxc/+zszHywg95+psZU662TniLiNOAEqs+xyzLzb7ro74PAb9ezvA64NTPP6bDWMHAz1fvziv9fHdbbHbi+rvcQcFZm/sxZTRExB7gLeCNwWmbeMu7ny4DzgVZd4/7xNZraoYOjbayrfwe8lWqsq/e3zXIlcDywBlgZEbdl5uZuawDPAUcB/306egK+AxySmS9FxNj8X+ii3vcz8+B6+eXAGcAFna6v+hf2eOBfp+G1AvxtZi4bv+4Ka3+M6oP/vR3WeJEqoJ+IiH2Bq4Ej6599dYowmqo3gGMz89lOa2TmUfV8bwSuzMzVETHEFOtusroR8W+Ak4BDgNcCNwH/fhprTLjeOqkXEW8CFmfmf5iOepl5I3BjPc+XqH53On2txwMrM/PSiPhN4HeAC7uodx5wTWbeGhFXAO8Ebh//uoFNwPuAD0+wTmYBl1KN+7cA+GL9PB3Z0XdVvTzWVZ2uL491FRFzgdmZ+URmrgMeBvaejhqZ+UJm/mS6esrMx9u2MDZS/YJ0U699oJsh4J+7XF8fBj5H9ddVV6+1/vHiiFgZEddGxKtL1yNA/RfeVOeab7VGZm7IzCfqh+PX+bKIuDciLouIiUYqmLS3uq+vRsQ3IuJXO3l9bT4I/FXb46nW3WR1fxn4XmZuzsz/B7wuInaZxhpbW2+d1HsPsDki7oiIGyPi56bjNUbEfGB/4L4uav2ALQOWDAOruuxtb6otSYAHgcOYQGa2MvOpiX5W13g4M9fWv9dz6v+DHdnRg2Oysa4WAiNtj0fqab2oMS31IuL1VH9tfLXbehHxjoj4e6qtjQfb5i+qFRHzgHdlZntP3fT2FPCGzFwCPAucNUHdqWo3NWWN+gPuCuDyetIDVP/ZDwX2BD7QQd3j69d3DvDZTnurHQd8pf6+ybqbrO6jwAERsWtE7AP8EtD+gdxNjcnWWyf19gTm1VscdwL/dZpe43uA/922K6iTWt8FDomIh4DfBf6yy94eotqDAdVW70TvyVTGP2+Tz6qt2tGDYzVV4o/ZPMnPhqh2MfWiRtf1ImI3ql/AkzNzY7f1MvPOzDyQarj6j3VR62zgU+NfZKf1MvPFzFxbT7uJanO+tHZTTWpcRfUX4v8ByMx1dY+jwJe20t+kdcd2UWXm94CXtrJlMGVvEfH2qkyuqes1WXdbrVtvJX8cuA24gGpLdKLdacU1plhvnfS0GvhmPdvtwJum6TWeyCu34Dqp9V+Av8zMN1H9/7i8bflO6v0JcGRE3AE8T/UHQqlOPqu2akcPjm8Dh0XErIh4G21jXWXmBmBTROxR/9W8N1Xi96JGV/Ui4jVU+1zPzszx43V1Uq9998MIsKGL17ov8NGIuB34pYi4rsveFrQtfxhbH59sq7ULTFojIs4DNmXmJ9umDTXob6q6C+qvu1P91byhtEbtg9T75dvrdtNbZt6cmYcBlwA/zOqWBl3XmGK9ddLT3cAB9Sy/Bvyo29dY/3G2R2b+U5e1BtkSRj/hlVsIxfUyc3VmfrDeuprFK/c4NPUIsE9EzKt/7zZl5gsd1AF2giFHIuJ3gFPYMtbVIcC/ZObdEfHrVAdoB4A/ycxbp6NGVAeLbwP2ozpg/MXMvKKLev8NOBP4YV3iM5n5+S7qvYdqN8ZovcxvZ+Zj3a6viPhuZu7f5bpbRvWfZh3VX0nLM7N9l1bT2udT7XbYjepkgHeW1AB+DDwG3Et1TOKJzDwpIj4MnE4Vtj8CTs9XHjOaqu5K4H5gPdXJKedn5p0dvL5ZVL8Pbx4Lnqbrboq6XwD2qJc/Myc4Y6yTGlOttw7qDQL/g+p4xCaqExme7LRevcyZwM9l5h93+Vr/LXAD1Yf8q4Azsu3stg7qHUl1gLxF9Vny5xO9J3Xtm6mCdB3VlthzVCclZEQcA/xhXefszPz21upMZYcPDknS9NrRd1VJkqaZwSFJKmJwSJKKGBySpCIGhySpiMEhzaCIWB4Rr217fEf99fCIuGnmOpO2zuCQZtZyqgHtAMgJBu2T+s0OPTquNB0i4lKqEUwfpx4CHrgpM3evf34x8HRm/llEXAIsBV5NNTT3efU8j1ENjf1uqtGAlwFHUF2s9dWIeDYzl0TE02N1257/NVTD4O9bT/q9zPy73r1iaXJucUiTiIgDqQaW25/qHgpvn2KRqzJzMfBm4M1R3ftkzKOZ+Rbgn4AP1ANDPgAcVw9MuDUXUN1XZDHVoIbXdPZqpOlhcEiTOxj4Sma+VA9HvXKK+Y+IiPup7qGymGrYmTFjQ7T8I9UQ2k0dCVwSEd8BVgC7RYR7CzRj/OWTptaa4Pv2abvAy/cYuQI4IDOfieouhO0DSr5Yfx2lGseoqQHg6PaxmKSZ5BaHNLn7gOMiYk5UtwQe26X0fESM3Whn7F4Jc6lCYXVUd3r7jQb117Llpj9bcwfVIJcAjNv9JW1zBoc0icz8e+BbVDfn+SwwdlD6Iqqhvb8JZD3vCNW9HH5AdXOlv23wFJ8DPh8Rk+0C+yNgz4j454j4PtWtR6UZ4+i4UoH62oo/y8y7ZroXaaa4xSFJKuIWhySpiFsckqQiBockqYjBIUkqYnBIkooYHJKkIv8f4U88UPVPu/IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "axc = sns.barplot(x=\"quantile\", y=\"customers\", data=customer_q, palette=five_thirty_eight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, only about 5% of customers have rated 5 or more videos, and only 25% of videos have been rated by 9+ customers.\n",
    "\n",
    "### Clean\n",
    "\n",
    "Let's filter out this long tail and remove any duplicate reviews (same product and customer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of records before removing the long tail =    3998345\n",
      "# of records after  removing the long tail =    1168955\n",
      "# of records after  removing duplicates    =    1168903\n"
     ]
    }
   ],
   "source": [
    "customers = customers[customers >= 5]\n",
    "products = products[products >= 10]\n",
    "\n",
    "print(\"# of records before removing the long tail = {:10d}\".format(reviews.shape[0]))\n",
    "reduced_df = reviews.merge(pd.DataFrame({'customer_id': customers.index})).merge(pd.DataFrame({'product_id': products.index}))\n",
    "print(\"# of records after  removing the long tail = {:10d}\".format(reduced_df.shape[0]))\n",
    "reduced_df = reduced_df.drop_duplicates(['customer_id', 'product_id'])\n",
    "print(\"# of records after  removing duplicates    = {:10d}\".format(reduced_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll recreate our customer and product lists since there are customers with more than 5 reviews, but all of their reviews are on products with less than 5 reviews (and vice versa)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "customers = reduced_df['customer_id'].value_counts()\n",
    "products = reduced_df['product_id'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll number each user and item, giving them their own sequential index.  This will allow us to hold the information in a sparse format where the sequential indices indicate the row and column in our ratings matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>product_title</th>\n",
       "      <th>customer</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27072354</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>5</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>10974</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16030865</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>5</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>481</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>44025160</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>5</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>31093</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>18602179</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>5</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>2096</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14424972</td>\n",
       "      <td>B008LOVIIK</td>\n",
       "      <td>5</td>\n",
       "      <td>Masterpiece: Inspector Lewis Season 5</td>\n",
       "      <td>34200</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customer_id  product_id  star_rating  \\\n",
       "0     27072354  B008LOVIIK            5   \n",
       "1     16030865  B008LOVIIK            5   \n",
       "2     44025160  B008LOVIIK            5   \n",
       "3     18602179  B008LOVIIK            5   \n",
       "4     14424972  B008LOVIIK            5   \n",
       "\n",
       "                           product_title  customer  product  \n",
       "0  Masterpiece: Inspector Lewis Season 5     10974      107  \n",
       "1  Masterpiece: Inspector Lewis Season 5       481      107  \n",
       "2  Masterpiece: Inspector Lewis Season 5     31093      107  \n",
       "3  Masterpiece: Inspector Lewis Season 5      2096      107  \n",
       "4  Masterpiece: Inspector Lewis Season 5     34200      107  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customer_index = pd.DataFrame({'customer_id': customers.index, 'customer': np.arange(customers.shape[0])})\n",
    "product_index = pd.DataFrame({'product_id': products.index, \n",
    "                              'product': np.arange(products.shape[0])})\n",
    "\n",
    "reduced_df = reduced_df.merge(customer_index).merge(product_index)\n",
    "reduced_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the feature dimension size whch will required for preparing the training and test data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(140344, 38385, 178729)\n"
     ]
    }
   ],
   "source": [
    "nb_customer = reduced_df['customer'].max() + 1\n",
    "nb_products = reduced_df['product'].max() + 1\n",
    "feature_dim = nb_customer + nb_products\n",
    "print(nb_customer, nb_products, feature_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trim down the data set to include only customer, product, star_rating which is all we need for the training algorithm to build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer</th>\n",
       "      <th>product</th>\n",
       "      <th>star_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10974</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>481</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>31093</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2096</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34200</td>\n",
       "      <td>107</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customer  product  star_rating\n",
       "0     10974      107            5\n",
       "1       481      107            5\n",
       "2     31093      107            5\n",
       "3      2096      107            5\n",
       "4     34200      107            5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "product_df = reduced_df[['customer', 'product', 'star_rating']]\n",
    "product_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare\n",
    "\n",
    "We will be using SageMaker's implementation of Factorization Machines (FM) for building a recommender system. The algorithm expects float32 tensors in protobuf whereas the data sets are pandas dataframe on disk. Most of the conversion effort is handled by the Amazon SageMaker Python SDK.\n",
    "\n",
    "The FM algorithm will utilize sparse input and since the data sets are dense matrix, it has to be converted a sparse matrix with one-hot encoded feature vectors with customers and products. Thus, each sample in the data set will be a wide boolean vector with 178729 feature space (140344 customer + 38385 products) with only two values set to 1 with respect to the customer and product.\n",
    "\n",
    "Following are the next steps\n",
    "\n",
    "1. Split the cleaned data set into train and test data sets.\n",
    "2. For each set, build a sparse matrix with one-hot encoded feature vectors (customer + products) and a label vector with star ratings.\n",
    "3. Convert both the sets to protobuf encoded files.\n",
    "4. Copy these files to an Amazon S3 bucket.\n",
    "5. Configure and run a Factorization Machines training job on Amazon SageMaker.\n",
    "6. Deploy the corresponding model to an endpoint.\n",
    "7. Run predictions on test data set and validate\n",
    "\n",
    "#### Split into Training and Test Data Sets\n",
    "\n",
    "Let's start by [splitting](https://docs.scipy.org/doc/numpy/reference/generated/numpy.split.html) in training, validation and test sets.  This will allow us to estimate the model's accuracy on videos our customers rated, but wasn't included in our training. We will use validation data set specifically for tuning model hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, validate_df, test_df = np.split(\n",
    "    product_df.sample(frac=1), \n",
    "    [int(.6*len(product_df)), int(.8*len(product_df))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of rows in the training data set   =     701341\n",
      "# of rows in the validation data set =     233781\n",
      "# of rows in the test data set       =     233781\n"
     ]
    }
   ],
   "source": [
    "print(\"# of rows in the training data set   = {:10d}\".format(train_df.shape[0]))\n",
    "print(\"# of rows in the validation data set = {:10d}\".format(validate_df.shape[0]))\n",
    "print(\"# of rows in the test data set       = {:10d}\".format(test_df.shape[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer</th>\n",
       "      <th>product</th>\n",
       "      <th>star_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1160169</th>\n",
       "      <td>76977</td>\n",
       "      <td>32390</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>482349</th>\n",
       "      <td>65653</td>\n",
       "      <td>473</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>661067</th>\n",
       "      <td>52655</td>\n",
       "      <td>1677</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>916814</th>\n",
       "      <td>14074</td>\n",
       "      <td>6523</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302666</th>\n",
       "      <td>92543</td>\n",
       "      <td>1814</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         customer  product  star_rating\n",
       "1160169     76977    32390            5\n",
       "482349      65653      473            5\n",
       "661067      52655     1677            5\n",
       "916814      14074     6523            5\n",
       "302666      92543     1814            4"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the feature dimensions by adding total number of (unique) customers and products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of customers =     140344\n",
      "# of products  =      38385\n",
      "# of features  =     178729\n"
     ]
    }
   ],
   "source": [
    "# get feature dimension\n",
    "all_df = pd.concat([train_df, validate_df, test_df])\n",
    "nb_customer = np.unique(all_df['customer'].values).shape[0]\n",
    "nb_products = np.unique(all_df['product'].values).shape[0]\n",
    "feature_dim = nb_customer + nb_products\n",
    "print(\"# of customers = {:10d}\".format(nb_customer))\n",
    "print(\"# of products  = {:10d}\".format(nb_products))\n",
    "print(\"# of features  = {:10d}\".format(feature_dim))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building Sparse One-Hot Encoded Matrix\n",
    "\n",
    "Our training matrix is now even sparser: Of all 183,833,321,511 values (1028559 rows * 178729 columns), only 2,057,118 are non-zero (1,028,559*2). In other words, the matrix is 99.99% sparse. Storing this as a dense matrix would be a massive waste of both storage and computing power. To avoid this, use a scipy.lil_matrix sparse matrix for features and a numpy array for ratings.\n",
    "\n",
    "Let's define a function that takes the data set and returns a sparse feature matrix and numpy array with ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_sparse_matrix(df, nb_rows, nb_customer, nb_products):\n",
    "    # dataframe to array\n",
    "    df_val = df.values\n",
    "\n",
    "    # determine feature size\n",
    "    nb_cols = nb_customer + nb_products\n",
    "    print(\"# of rows = {}\".format(str(nb_rows)))\n",
    "    print(\"# of cols = {}\".format(str(nb_cols)))\n",
    "\n",
    "    # extract customers and ratings\n",
    "    df_X = df_val[:, 0:2]\n",
    "    # Features are one-hot encoded in a sparse matrix\n",
    "    X = lil_matrix((nb_rows, nb_cols)).astype('float32')\n",
    "    df_X[:, 1] = nb_customer + df_X[:, 1]\n",
    "    coords = df_X[:, 0:2]\n",
    "    X[np.arange(nb_rows), coords[:, 0]] = 1\n",
    "    X[np.arange(nb_rows), coords[:, 1]] = 1\n",
    "\n",
    "    # create label with ratings\n",
    "    Y = df_val[:, 2].astype('float32')\n",
    "\n",
    "    # validate size and shape\n",
    "    print(X.shape)\n",
    "    print(Y.shape)\n",
    "    assert X.shape == (nb_rows, nb_cols)\n",
    "    assert Y.shape == (nb_rows, )\n",
    "\n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Convert training data set to one-hot encoded sparse matrix\n",
      "# of rows = 701341\n",
      "# of cols = 178729\n",
      "(701341, 178729)\n",
      "(701341,)\n",
      "Convert validation data set to one-hot encoded sparse matrix\n",
      "# of rows = 233781\n",
      "# of cols = 178729\n",
      "(233781, 178729)\n",
      "(233781,)\n",
      "Convert test data set to one-hot encoded sparse matrix\n",
      "# of rows = 233781\n",
      "# of cols = 178729\n",
      "(233781, 178729)\n",
      "(233781,)\n"
     ]
    }
   ],
   "source": [
    "print(\"Convert training data set to one-hot encoded sparse matrix\")\n",
    "train_X, train_Y = convert_sparse_matrix(train_df, train_df.shape[0], nb_customer, nb_products)\n",
    "print(\"Convert validation data set to one-hot encoded sparse matrix\")\n",
    "validate_X, validate_Y = convert_sparse_matrix(validate_df, validate_df.shape[0], nb_customer, nb_products)\n",
    "print(\"Convert test data set to one-hot encoded sparse matrix\")\n",
    "test_X, test_Y = convert_sparse_matrix(test_df, test_df.shape[0], nb_customer, nb_products)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert to Protobuf format and Upload to S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Sagemaker's utility function [`write_spmatrix_to_sparse_tensor`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/amazon/common.py) to convert scipy sparse matrix to protobuf format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_as_protobuf(X, Y, bucket, key):\n",
    "    \"\"\"Converts features and predictions matrices to recordio protobuf and\n",
    "       writes to S3\n",
    "\n",
    "    Args:\n",
    "        X:\n",
    "          2D numpy matrix with features\n",
    "        Y:\n",
    "          1D numpy matrix with predictions\n",
    "        bucket:\n",
    "          s3 bucket where recordio protobuf file will be staged\n",
    "        prefix:\n",
    "          s3 url prefix to stage prepared data to use for training the model\n",
    "        key:\n",
    "          protobuf file name to be staged\n",
    "\n",
    "    Returns:\n",
    "        s3 url with key to the protobuf data\n",
    "    \"\"\"\n",
    "    buf = io.BytesIO()\n",
    "    smac.write_spmatrix_to_sparse_tensor(buf, X, Y)\n",
    "    buf.seek(0)\n",
    "    obj = '{}'.format(key)\n",
    "    boto3.resource('s3').Bucket(bucket).Object(obj).upload_fileobj(buf)\n",
    "    return 's3://{}/{}'.format(bucket, obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data set in protobuf format uploaded at s3://sagemaker-us-east-1-835319576252/prepare/train/train.protobuf\n",
      "Validation data set in protobuf format uploaded at s3://sagemaker-us-east-1-835319576252/prepare/validate/validate.protobuf\n"
     ]
    }
   ],
   "source": [
    "s3_train_path = save_as_protobuf(train_X, train_Y, bucket, 'prepare/train/train.protobuf')\n",
    "print(\"Training data set in protobuf format uploaded at {}\".format(s3_train_path))\n",
    "s3_val_path = save_as_protobuf(validate_X, validate_Y, bucket, 'prepare/validate/validate.protobuf')\n",
    "print(\"Validation data set in protobuf format uploaded at {}\".format(s3_val_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will chunk the test data to avoid the payload size issues when performing batch predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chunk(x, batch_size):\n",
    "    \"\"\"split array into chunks of batch_size\n",
    "    \"\"\"\n",
    "    chunk_range = range(0, x.shape[0], batch_size)\n",
    "    chunks = [x[p: p + batch_size] for p in chunk_range]\n",
    "    return chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_0.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_1.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_2.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_3.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_4.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_5.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_6.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_7.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_8.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_9.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_10.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_11.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_12.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_13.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_14.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_15.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_16.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_17.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_18.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_19.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_20.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_21.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_22.protobuf\n",
      "s3://sagemaker-us-east-1-835319576252/prepare/test/test_23.protobuf\n"
     ]
    }
   ],
   "source": [
    "test_x_chunks = chunk(test_X, 10000)\n",
    "test_y_chunks = chunk(test_Y, 10000)\n",
    "N = len(test_x_chunks)\n",
    "for i in range(N):\n",
    "    test_data = save_as_protobuf(\n",
    "        test_x_chunks[i],\n",
    "        test_y_chunks[i],\n",
    "        bucket,\n",
    "        \"prepare/test/test_\" + str(i) + \".protobuf\")\n",
    "    print(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training\n",
    "\n",
    "Once we have the data preprocessed and available in the correct format for training, the next step is to actually train the model using the data. We'll use the Amazon SageMaker Python SDK to kick off training and monitor status until it is completed. In this example that takes between 4-7 minutes for 3-10 epochs. \n",
    "\n",
    "First, let's get the Sagemaker Factorization Machine container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "container = get_image_uri(boto3.Session().region_name, 'factorization-machines')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next kick off the base estimator, making sure to pass in the necessary hyperparameters. Notice:\n",
    "\n",
    "- `feature_dim` is set to 178729, which is the number of customers + products in the training data set.\n",
    "- `predictor_type` is set to 'regressor' since we are trying to predict the rating\n",
    "- `mini_batch_size` is set to 200. This value can be tuned for relatively minor improvements in fit and speed, but selecting a reasonable value relative to the dataset is appropriate in most cases.\n",
    "- `num_factors` is set to 64. Factorization machines find a lower dimensional representation of the interactions for all features. Making this value smaller provides a more parsimonious model, closer to a linear model, but may sacrifice information about interactions. Making it larger provides a higher-dimensional representation of feature interactions, but adds computational complexity and can lead to overfitting. In a practical application, time should be invested to tune this parameter to the appropriate value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2 µs, sys: 0 ns, total: 2 µs\n",
      "Wall time: 3.81 µs\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "\n",
    "output_location = 's3://{}/train/'.format(bucket)\n",
    "s3_train_path = 's3://{}/prepare/train/train.protobuf'.format(bucket)\n",
    "s3_val_path = 's3://{}/prepare/validate/validate.protobuf'.format(bucket)\n",
    "\n",
    "fm = sagemaker.estimator.Estimator(container,\n",
    "                                   role, \n",
    "                                   train_instance_count=1, \n",
    "                                   train_instance_type='ml.c5.4xlarge',\n",
    "                                   output_path=output_location,\n",
    "                                   sagemaker_session=sess)\n",
    "\n",
    "fm.set_hyperparameters(feature_dim=feature_dim,\n",
    "                      predictor_type='regressor',\n",
    "                      mini_batch_size=200,\n",
    "                      num_factors=512,\n",
    "                      bias_lr=0.02,\n",
    "                      epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm.fit({'train': s3_train_path,'test': s3_val_path}, wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker built-in algorithms automatically compute and emit a variety of model training, evaluation, and validation metrics that can be captured from Cloudwatch using Sagemaker SDK. Since we are using FM built-in algorithm with predictor type as `regressor`, we can capture RMSE (root-mean-square error) of the model that measures the differences between the predicted values and the actual values.\n",
    "\n",
    "Let's capture the RMSE of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job_name = fm._current_job_name\n",
    "metric_name = 'train:rmse:epoch'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{u'TrainingJobName': u'factorization-machines-2020-10-27-00-41-14-792', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'ba874c9a-4ebd-415d-8538-b3819757aaac', 'HTTPHeaders': {'x-amzn-requestid': 'ba874c9a-4ebd-415d-8538-b3819757aaac', 'date': 'Tue, 27 Oct 2020 00:43:15 GMT', 'content-length': '4962', 'content-type': 'application/x-amz-json-1.1'}}, u'TrainingJobArn': u'arn:aws:sagemaker:us-east-1:835319576252:training-job/factorization-machines-2020-10-27-00-41-14-792', u'InputDataConfig': [{u'CompressionType': u'None', u'ChannelName': u'test', u'DataSource': {u'S3DataSource': {u'S3DataType': u'S3Prefix', u'S3DataDistributionType': u'FullyReplicated', u'S3Uri': u's3://sagemaker-us-east-1-835319576252/prepare/validate/validate.protobuf'}}, u'RecordWrapperType': u'None'}, {u'CompressionType': u'None', u'ChannelName': u'train', u'DataSource': {u'S3DataSource': {u'S3DataType': u'S3Prefix', u'S3DataDistributionType': u'FullyReplicated', u'S3Uri': u's3://sagemaker-us-east-1-835319576252/prepare/train/train.protobuf'}}, u'RecordWrapperType': u'None'}], u'AlgorithmSpecification': {u'TrainingInputMode': u'File', u'TrainingImage': u'382416733822.dkr.ecr.us-east-1.amazonaws.com/factorization-machines:1', u'EnableSageMakerMetricsTimeSeries': False, u'MetricDefinitions': [{u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train rmse <loss>=(\\\\S+)', u'Name': u'train:rmse:epoch'}, {u'Regex': u'#progress_metric: host=\\\\S+, completed (\\\\S+) %', u'Name': u'train:progress'}, {u'Regex': u'#quality_metric: host=\\\\S+, test binary_f_1.000 <score>=(\\\\S+)', u'Name': u'test:binary_f_beta'}, {u'Regex': u'#quality_metric: host=\\\\S+, test mse <loss>=(\\\\S+)', u'Name': u'test:mse'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train binary_classification_accuracy <score>=(\\\\S+)', u'Name': u'train:binary_classification_accuracy:batch'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train mse <loss>=(\\\\S+)', u'Name': u'train:mse:batch'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train absolute_loss <loss>=(\\\\S+)', u'Name': u'train:absolute_loss:epoch'}, {u'Regex': u'#quality_metric: host=\\\\S+, test absolute_loss <loss>=(\\\\S+)', u'Name': u'test:absolute_loss'}, {u'Regex': u'#quality_metric: host=\\\\S+, train mse <loss>=(\\\\S+)', u'Name': u'train:mse'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train binary_f_1.000 <score>=(\\\\S+)', u'Name': u'train:binary_f_beta:batch'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train binary_classification_cross_entropy <loss>=(\\\\S+)', u'Name': u'train:binary_classification_cross_entropy:epoch'}, {u'Regex': u'#quality_metric: host=\\\\S+, train binary_f_1.000 <score>=(\\\\S+)', u'Name': u'train:binary_f_beta'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train mse <loss>=(\\\\S+)', u'Name': u'train:mse:epoch'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train rmse <loss>=(\\\\S+)', u'Name': u'train:rmse:batch'}, {u'Regex': u'#quality_metric: host=\\\\S+, train binary_classification_cross_entropy <loss>=(\\\\S+)', u'Name': u'train:binary_classification_cross_entropy'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train binary_f_1.000 <score>=(\\\\S+)', u'Name': u'train:binary_f_beta:epoch'}, {u'Regex': u'#quality_metric: host=\\\\S+, train rmse <loss>=(\\\\S+)', u'Name': u'train:rmse'}, {u'Regex': u'#quality_metric: host=\\\\S+, test rmse <loss>=(\\\\S+)', u'Name': u'test:rmse'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, train binary_classification_accuracy <score>=(\\\\S+)', u'Name': u'train:binary_classification_accuracy:epoch'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train absolute_loss <loss>=(\\\\S+)', u'Name': u'train:absolute_loss:batch'}, {u'Regex': u'#quality_metric: host=\\\\S+, train binary_classification_accuracy <score>=(\\\\S+)', u'Name': u'train:binary_classification_accuracy'}, {u'Regex': u'#quality_metric: host=\\\\S+, test binary_classification_cross_entropy <loss>=(\\\\S+)', u'Name': u'test:binary_classification_cross_entropy'}, {u'Regex': u'#quality_metric: host=\\\\S+, train absolute_loss <loss>=(\\\\S+)', u'Name': u'train:absolute_loss'}, {u'Regex': u'#throughput_metric: host=\\\\S+, train throughput=(\\\\S+) records/second', u'Name': u'train:throughput'}, {u'Regex': u'#quality_metric: host=\\\\S+, test binary_classification_accuracy <score>=(\\\\S+)', u'Name': u'test:binary_classification_accuracy'}, {u'Regex': u'#quality_metric: host=\\\\S+, epoch=\\\\S+, batch=\\\\S+ train binary_classification_cross_entropy <loss>=(\\\\S+)', u'Name': u'train:binary_classification_cross_entropy:batch'}]}, u'HyperParameters': {u'predictor_type': u'regressor', u'epochs': u'10', u'feature_dim': u'178729', u'bias_lr': u'0.02', u'num_factors': u'512', u'mini_batch_size': u'200'}, u'RoleArn': u'arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881', u'CreationTime': datetime.datetime(2020, 10, 27, 0, 41, 14, 990000, tzinfo=tzlocal()), u'SecondaryStatusTransitions': [{u'Status': u'Starting', u'StatusMessage': u'Preparing the instances for training', u'StartTime': datetime.datetime(2020, 10, 27, 0, 41, 14, 990000, tzinfo=tzlocal())}], u'EnableManagedSpotTraining': False, u'TrainingJobStatus': u'InProgress', u'EnableInterContainerTrafficEncryption': False, u'OutputDataConfig': {u'KmsKeyId': u'', u'S3OutputPath': u's3://sagemaker-us-east-1-835319576252/train/'}, u'StoppingCondition': {u'MaxRuntimeInSeconds': 86400}, u'EnableNetworkIsolation': False, u'ResourceConfig': {u'VolumeSizeInGB': 30, u'InstanceCount': 1, u'InstanceType': u'ml.c5.4xlarge'}, u'LastModifiedTime': datetime.datetime(2020, 10, 27, 0, 42, 46, 366000, tzinfo=tzlocal()), u'SecondaryStatus': u'Starting'}\n",
      "Reminder: the training job has not been completed.\n"
     ]
    }
   ],
   "source": [
    "# run this cell to check current status of training job\n",
    "fm_training_job_result = sm.describe_training_job(TrainingJobName=training_job_name)\n",
    "print(fm_training_job_result)\n",
    "\n",
    "status = fm_training_job_result['TrainingJobStatus']\n",
    "if status != 'Completed':\n",
    "    print('Reminder: the training job has not been completed.')\n",
    "else:\n",
    "    print('The training job is completed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2020-10-27 00:44:04 Starting - Preparing the instances for training\n",
      "2020-10-27 00:44:04 Downloading - Downloading input data\n",
      "2020-10-27 00:44:04 Training - Training image download completed. Training in progress.....................................\n",
      "2020-10-27 00:48:01 Uploading - Uploading generated training model........\n",
      "2020-10-27 00:48:48 Completed - Training job completed\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "fm.latest_training_job.wait(logs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait for the job ^^ above ^^ to complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric_name</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>train:rmse:epoch</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.185032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>train:rmse:epoch</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.998977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>train:rmse:epoch</td>\n",
       "      <td>120.0</td>\n",
       "      <td>0.866597</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        metric_name  timestamp     value\n",
       "0  train:rmse:epoch        0.0  1.185032\n",
       "1  train:rmse:epoch       60.0  0.998977\n",
       "2  train:rmse:epoch      120.0  0.866597"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# plug-in the training job name and metrics to be captured\n",
    "metrics_dataframe = TrainingJobAnalytics(training_job_name=training_job_name,\n",
    "                                         metric_names=[metric_name]).dataframe()\n",
    "metrics_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAFCCAYAAAAt72H5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi41LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvSM8oowAAG3RJREFUeJzt3XGUnXV95/H3ZDIlHQwZAlkqB4vVyrcWoYg0WjchWlarOVRbTRUVC4RQrWGlSLV1EQHxtBy1uy5FXUtFqEurgUoXgrAcdhsSS41QRcQDX0N1OXswsJHMSGIIyWTu/nGf1MuQzDyZ5Dd3npv36xzO3Of33Oe53+Gbe/LJ7/7u8/S1Wi0kSZIklTOr2wVIkiRJvc7QLUmSJBVm6JYkSZIKM3RLkiRJhRm6JUmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVNjsUieOiAFgDXA8sCIzb+rY98vAF4EWsBM4NzP/T0T8PHAdcDTwILAyM8dK1ShJkiRNh5Iz3aPAMuDTe9g3DLw5M08FPglcXI0vB+7LzMXAGPCGgvVJkiRJ06LYTHdmtoCNEbGnfU92bO6gHdABFgOXV49XA0uAr5WqUZIkSZoOxUJ3HRExB7gM+INqaD7tWXCAkWq7jtaBrUySJEnao76pHNS10B0R/cB/B/5rZj5cDQ8DQ8DjwDxgc93zbdq05YDXqPKGhgYZGdnW7TI0Rfav2exfc9m7ZrN/zbVgwdwpH9vNq5d8FlibmX/fMbYWWFo9Xgqsm/aqJEmSpAOs6Ex3RKwCTgG2RsRC2jPXNwP/DjgL+EZEvAX4l8y8iPYVTa6LiLXAQ8DtJeuTJEmSpkPR0J2Zb9vbLmDOHp6/DdjbMZIkSVIjeXMcSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkFWboliRJkgrrmdC9c3Ss2yVIkiRJe9QzofvSa79p8JYkSdKM1DOhe/NT23li2FuqSpIkaebpmdA9/7A5HHX4YLfLkCRJkp6jZ0L35csXMjC7Z34dSZIk9ZCeSakGbkmSJM1UJlVJkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkFWboliRJkgozdEuSJEmFGbolSZKkwgzdkiRJUmGGbkmSJKkwQ7ckSZJUmKFbkiRJKszQLUmSJBU2u9SJI2IAWAMcD6zIzJvG7b8ZWAxcmZmfqsbOBi4B/i9AZr6mVH2SJEnSdCkWuoFRYBnwnr3sPx94HXDkuPHP7Q7hkiRJUi8otrwkM1uZuXGC/Y/tZdeKiPh6RPxRodIkSZKkaVVypnsq/gH4Eu26bo2Ib2TmN+ocODQ0WLQwldHfP8veNZj9azb711z2rtns38FpRoXuzBypHu6KiH8AXg7UCt0jI9uK1aVyhoYG7V2D2b9ms3/NZe+azf4114IFc6d87Iy6eklEzKt+9gGnAhu6W5EkSZK0/4rOdEfEKuAUYGtELAQ2AzdnZkbEVcBpwEBEnJCZZwEXRcTrgRawNjPvKlmfJEmSNB36Wq1Wt2s4EFqbNm3pdg2aAj9iazb712z2r7nsXbPZv+aqlpf0TeXYGbW8RJIkSepFhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkFWboliRJkgozdEuSJEmFGbolSZKkwgzdkiRJUmGGbkmSJKkwQ7ckSZJUmKFbkiRJKszQLUmSJBVm6JYkSZIKM3RLkiRJhRm6JUmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYXNLnXiiBgA1gDHAysy86Zx+28GFgNXZuanqrGfB64DjgYeBFZm5lipGiVJkqTpUHKmexRYBnx6L/vPB/543Nhy4L7MXAyMAW8oV54kSZI0PYqF7sxsZebGCfY/tofhxcDq6vFq4NQStUmSJEnTqdjykimaDwxXj0eq7VqGhgaLFKSy+vtn2bsGs3/NZv+ay941m/07OM200D0MDAGPA/OAzXUPHBnZVqomFTQ0NGjvGsz+NZv9ay5712z2r7kWLJg75WNn2tVL1gJLq8dLgXVdrEWSJEk6IIrOdEfEKuAUYGtELKQ9c31zZmZEXAWcBgxExAmZeRbwReC6iFgLPATcXrI+SZIkaTr0tVqtbtdwILQ2bdrS7Ro0BX7E1mz2r9nsX3PZu2azf81VLS/pm8qxM215iSRJktRzDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkFWboliRJkgozdEuSJEmFGbolSZKkwgzdkiRJUmGGbkmSJKkwQ7ckSZJUmKFbkiRJKszQLUmSJBU2e7InRMSJwIeAY4F+oA9oZearC9cmSZIk9YRJQzfwd8AlwLeBsbLlSJIkSb2nTujemplfLV6JJEmS1KP2Groj4vXVw3+OiGuAW4Bndu/PzDsL1yZJkiT1hIlmut8xbvstHY9bgKFbkiRJqmGvoTszz5nOQiRJkqReNeklAyPitogY6tg+PCJuKVuWJEmS1DvqXKf76Mwc2b2RmcPAC8qVJEmSJPWWOqF7R0Qct3sjIgLYWa4kSZIkqbfUuWTghcBtEbGB9o1xXgT8ftGqJEmSpB4yaejOzHsi4nggqqGHM9OZbkmSJKmmOl+k7AOWAe+t/ntrNSZJkiSphjrLS64GjgBuqLbfAZwKvK9UUZIkSVIvqRO6F2Xmr3Vs3xoR3ylVkCRJktRr6l695KTdGxFxArCjXEmSJElSb6kz0/0+4EvVOu4+YBRYXrQqSZIkqYfUuXrJvcAJETGv2v5J8aokSZKkHjJp6I6II4BLgUVAKyK+DnwsM5+c5LgBYA1wPLAiM28at/904GKgBVyQmfdGxGtof2FzQ/W0MzLz8X36jSRJkqQZps7ykq8AtwBX0A7IZwCrgNMmOW6U9qUG3zN+R0T0V+c7FTiseo1F1e6bM/P8OsVLkiRJTVAndB+VmVd1bF8dEX8w2UGZ2QI2tu8a/xwvAb6fmVuALRExEBFzqn2nV1/cvBv4SHUeSZIkqbHqhO7bImIl7dnoFvB2YHVE/BxAZk7lSibzgeGO7ZFq7D7ad77cCXyheq0v1znh0NDgFMpQt/X3z7J3DWb/ms3+NZe9azb7d3CqE7rfXv3843Hj76Adwl80hdcdBoY6tucBmzNz++6BiLgRWELN0D0ysm0KZajbhoYG7V2D2b9ms3/NZe+azf4114IFc6d8bJ2rl/zSlM++dxuA4yLiUGAuMJqZ2yNiXsfVUZbwsy9USpIkSY1V5+olC4CPAkdn5lsj4leAhZn5NzWOXQWcAmyNiIXAZtpflMyIuAy4i/Zs+YXVIWdExHnA08C/Ah+Zwu8kSZIkzSh1lpdcD1wLXFJtPwLcCEwaujPzbRPsu4X2VVE6xz4PfL5GTZIkSVJj1LkN/ILqGttjAJk5CuwqWpUkSZLUQ+qE7uGIeAHtZSBExOuA/1e0KkmSJKmH1Fle8j7gr2l/8fFh2lceObNoVZIkSVIPqXP1kkeA34qI5wGzMvOp8mVJkiRJvaPO8hIi4uTM3Lo7cEfEyWXLkiRJknpHrdAN/OEk25IkSZL2olbozszzJtqWJEmStHd1bo4zh/aXKY/NzAsi4sXAizPzzuLVSZIkST2gzkz39dXzTqu2NwKfKFaRJEmS1GPqhO4XZ+angJ0AmbmtbEmSJElSb6kTurdFxDx+dnOclwFbi1YlSZIk9ZA6N8e5CPgfwAsj4g7gl4B3Fq1KkiRJ6iF1bo5zb3Xr9wD6gIczc2fxyiRJkqQeMenykog4A5iTmQ8Cvwt8JSIWFq9MkiRJ6hF11nR/ODO3RMRvAq8GPg1cXbYsSZIkqXfUCd27n/Nm4AuZuRYYKFeSJEmS1FvqhO77I2INcDrwtYiYS3UlE0mSJEmTqxO6zwI+ALwiM39Ke5b7nKJVSZIkST2kziUDAY4AXh4R/R1j3ylQjyRJktRz6oTuW4HtwIPAWDXm8hJJkiSppjqh+5jM/LXilUiSJEk9qk7o/kpEvB24DdixezAzd+z9EEmSJEm71QndTwKfB0ZoLy/po7285EUF65IkSZJ6Rp3Q/SfAL2fmj0sXI0mSJPWiOpcMfIiOZSWSJEmS9k2dme7ZwMMRcQ/PXtP9zmJVSZIkST2kTuj+s+JVSJIkST1swtAdEbOAj2bmadNUjyRJktRzJlzTnZljwI6IWDBN9UiSJEk9Z1/WdK8Dtu0edE23JEmSVE+d0P3xqZw4IgaANcDxwIrMvGnc/tOBi2lf8/uCzLy3Ws7yGeBlwI+AszPz6am8viRJkjRTTBq6M/PuKZ57FFgGvGf8jojoB64ATgUOA74CLALeCIxl5uKI+CBwDvDZKb6+JEmSNCPUuU43EfFXE23vSWa2MnPjXna/BPh+Zm7JzMeAgYiYAywGVlfPWU07lEuSDrCdo2M8+vhT7Bwd63YpknRQqLO8BNq3gZ9oe1/NB4Y7tkeqsc7x3WOSpANo5+gYl177TTZveYb5cw/h8uULGZhdaw5GkjRFtUJ3Zv7LRNtTMAwMdWzPAzaPG989VsvQ0OB+lqRu6O+fZe8azP4106OPP8XmLc+wY+cuNm95hm2jYxx75PO6XZb2ge+9ZrN/B6dJQ3dEnAh8CDgW6Af6gFZmvno/XncDcFxEHArMBUYzc3tErAWWAndUP9fVPeHIyLbJn6QZZ2ho0N41mP1rpsHZs5g/95B/m+kenD3LPjaM771ms3/NtWDB3CkfW2em+++AS4BvA/u0+C8iVgGnAFsjYiHtmeubMzMj4jLgLtpXL7mwOuR24E3V5Qk3Amfvy+tJkiY3MHsWly9fyLbRMQZnz3JpiSRNg75WqzXhEyJifWa+cprqmarWpk1bul2DpsB/7Teb/Ws2+9dc9q7Z7F9zVTPdfVM5ts5M9z9HxDXALcAzuwcz886pvKAkSZJ0sKkTuudVP9/SMdYCDN2SJElSDXVujnPOdBQiSZIk9aq9hu6IOD8zr46IP9vT/sz8T+XKkiRJknrHRDPdj1Y/czoKkSRJknrVXkN3Zt5a/bx++sqRJEmSek+dm+ME8OfAS4FDdo9n5osK1iVJkiT1jDpXL7kW+CPgGmARcA5waMmiJEmSpF5S5zZkh2TmvUB/Zj6ZmZ8C3ly4LkmSJKln1JnpfjoiBoAHIuIS4PGax0mSJEmi3kz3WUA/sLL6+VJgWcmiJEmSpF4y4Yx1RMwCPpaZZwLbgcumoyhJkiSpl0w4052ZY8AxETE4TfVIkiRJPafO2uxNwLci4g5g2+5B70gpSZIk1VMndK8HVo8bO7JALZIkSVJPqhO635mZJ3cORMT9wF+UKUmSJEnqLXsN3RFxLrACOC4i7unYNRf4bunCJEmSpF4x0Uz3TcD/Aj4OXNwxviUzNxetSpIkSeohew3dmfkT4CfAmdNXjiRJktR76twcR5IkSdJ+MHRLkiRJhRm6JUmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYbNLnjwizgPOAXYAyzPzBx37VgJnVvsuzMxvRcRrgBuADdXTzsjMx0vWKEmSJJVWLHRHxHxgBfDvgZcDVwJvq/YdBbwLWAQcCXwZeG116M2ZeX6puiRJkqTpVnKmeyGwJjNHgXsjIjr2vRD4XmbuAp6IiGMi4pBq3+kRcRJwN/CRzGwVrFGSJEkqrmTong8Md2x3rh9/BHhFRAwCxwDHAocD9wEB7AS+ALyd9iz4pIaGBg9AyZpu/f2z7F2D2b9ms3/NZe+azf4dnEqG7mHgxI7tXbsfZOaTEXElcDvwKPAA8ONqVhyAiLgRWELN0D0ysu1A1KxpNjQ0aO8azP41m/1rLnvXbPavuRYsmDvlY0tevWQ9sCQi+iPiZH725UgAMnNVZi4BLgcezszRiJjX8ZQl44+RJEmSmqjYTHdmbo6I64F1tJeLnBsRZwM/zMy7I+IG4Pm0Z8RXVoedUV3x5GngX4GPlKpPkiRJmi59rVZPfE+xtWnTlm7XoCnwI7Zms3/NZv+ay941m/1rrmp5Sd9UjvXmOJIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkFWboliRJkgozdEuSJEmFGbolSZKkwgzdkiRJUmGGbkmSJKkwQ7ckSZJUmKFbkiRJKszQLUmSJBVm6JYkSZIKM3RLkiRJhRm6JUmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYXNLnnyiDgPOAfYASzPzB907FsJnFntuzAzvxURs4DPAC8DfgScnZlPl6xRkiRJKq3YTHdEzAdWAKcCHwSu7Nh3FPAuYBHwNuAvql1vBMYyczFwH+3ALkmSJDVayeUlC4E1mTmamfcC0bHvhcD3MnNXZj4BHBMRhwCLgdXVc1bTDuySJElSo5VcXjIfGO7Y7gz4jwCviIhB4BjgWODwcceMVNu1DA0N7lex6o7+/ln2rsHsX7PZv+ayd81m/w5OJUP3MHBix/au3Q8y88mIuBK4HXgUeAD4cXXMUPW0ecDmui82MrJtf+tVFwwNDdq7BrN/zWb/msveNZv9a64FC+ZO+diSy0vWA0sioj8iTgY2dO7MzFWZuQS4HHg4M0eBtcDS6ilLgXUF65MkSZKmRbGZ7szcHBHX0w7OO4FzI+Js4IeZeXdE3AA8n/bs9srqsNuBN0XEOmAjcHap+iRJkqTp0tdqtbpdw4HQ2rRpS7dr0BT4EVuz2b9ms3/NZe+azf41V7W8pG8qx3pzHEmSJKkwQ7ckSZJUmKFbkiRJKszQLUmSJBVm6JYkSZIKM3RLkiRJhRm6JUmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSZIkTWLn6Nh+HW/oliRJkiawc3SMS6/95n6dw9AtSZIkTeCJ4W1sfmr7fp3D0C1JkiRN4KjDB5l/2Jz9OoehW5IkSZrAwOxZXL584X6dw9AtSZIkTWJg9v7FZkO3JEmSVJihW5IkSSrM0C1JkiQVZuiWJEmSCjN0S5IkSYUZuiVJkqTCDN2SJElSYYZuSZIkqTBDtyRJklRYX6vV6nYNB0JP/BKSJEma8fqmctDsA11Fl0zpl5ckSZKmg8tLJEmSpMIM3ZIkSVJhhm5JkiSpMEO3JEmSVJihW5IkSSrM0C1JkiQV1vhLBkbEecA5wA5geWb+oMslaQIR8RvAf6bdr63Au2j/OfwSMBe4KzMv61qBmlRELALWAQuqIXvXEBGxEPg4MAB8Dfgi9q8RIuJq4GTak2V/CqwHrgOOBh4EVmbmWNcK1LNExACwBjgeWJGZN0XEkezh/RYRpwMX077nyAWZeW9Xita/2Uv/rgFeRvs9eHVmfql6bu0c2uiZ7oiYD6wATgU+CFzZ3YpUw6PAaZm5BLgVWAn8CXBtZi4Cfj0ifrWbBWpSFwL3VY/tXUNExCHApcDvZOZrM/OT2L9GiIiXAC/NzFcDvwd8DFgO3JeZi4Ex4A1dLFHPNQosAz7dMfac91tE9ANXAK+n3dv/Mu2Vak/21L9PZOZv0M6cF0fE7H3NoY0O3cBCYE1mjlb/MoxuF6SJZeaPMnNbtbmD9h/sRcDqamw17T+8moGqGZmvAz+thuxdc7wKeBq4KSLujIgTsX9N8QTwdETMBoaATcBi7N2MlZmtzNw4bnhP77eXAN/PzC2Z+RgwEBFzprFU7cGe+peZG6qHO4BdtD+Z2Kcc2vTQPR8Y7thu+u9z0IiII4D3AV8ADs3Mp6tdI7T7qhkmImbR7tnnOobtXXMcDfwq7dm0C4DPYv+aYgvtTwkTuBP4JM/++8/eNcOe3m/jc4y9nPkuAlZl5i72MYc2PaQO0/5X/267ulWI6ouIQeBG4P2Z+WNgW8e/7OcBm7tWnCbyTuCWzNzeMWbvmmMY+KfM/GlmPkS7X/avGV4HHEF7VvRk4DM8++8/e9cMe3q/jc8x9nIGi4hltD81vKIa2qcc2vTQvR5YEhH9EXEysGGyA9Rd1cejXwb+MjPvqYbXAUurx2+stjXznAAsi4g7gBOBv8XeNcl6ICJiVkT8ArAd+9cUs4DN1RclnwIOBdbys94txd41wZ7ebxuA4yLi0Op9OTpuYkMzRES8hvanve/u+NLyPuXQvlarVbbKwiLivcDvAzuBczPzkS6XpAlExLuBvwTur4Zuo/0N/L+h/Y3u/52ZH+1OdaorItbQ/pJJH/auMSJiOe1v2Q8AHwIewv7NeNWX7a4DXgjMof3lrpursV+g3cc/9OolM0tErAJOoX2lrjtoLwt6zvstIt4EfJj2GuELM3N9dypWpz307620l3qNVE/5vczctC85tPGhW5IkSZrpmr68RJIkSZrxDN2SJElSYYZuSZIkqTBDtyRJklSYoVuSJEkqzNAtSTNMRAxFxIrq8Zsi4v2FXuekiPgPJc4tSXq22d0uQJL0HEPACuCvM/OWgq9zEvArwF0FX0OShNfplqQZJyJuAH6H9t3NbgCOyMw/jYjraN+o4ZXA82jfkOGjwEuBT2TmX1XHX1wdfwjtu79eExGvBa6ifZvinwKnAj+snrMRuJD2Lag/XI09CrwrM7fWed3qbm0XV7/CLwJfzsxLS/z/kaQmcnmJJM08FwPfzcyTgE3j9v1cZv468N+AVbQD8KuqY4iINwALquecAqyIiOcDHwDeX51zaWbuoh2cv5iZJ2XmPwJ3Z+Yrq+f8E3Bu3detvAo4GzgB+K2IOOXA/O+QpOZzeYkkNcvq6ud3gW9n5jBARPRFxADwOuC3q5lnaM9evxi4B/hERFxPOzTvyS9GxI3AUcAgz152MtnrAqzLzMeq8a8Ci4D79ueXlaReYeiWpGbZUf0c63i8e7sf6AMuycy/HXfc1yPiduC3gfV7mYW+Crg0M9dExDLg9H14XYDO9YqtcduSdFAzdEvSzLMFmDvFY+8CLoqIr2bm9ogI2uuzj87M+4H7I+I3gRfs4XUOAzZGxCzg3cDwPr724og4mvaSmLcA/3GKv4Mk9RzXdEvSDJOZTwIPRMQDwIJ9PPZrwP8E7o2IB4HP0Z6J/kBEfK8658PAd4B/BBZGxLerL1peAdwOrKf9Jct9tR64DngQuDMzXVoiSRWvXiJJ2m/VGvL3ZuYZ3a5FkmYiZ7olSZKkwpzpliRJkgpzpluSJEkqzNAtSZIkFWboliRJkgozdEuSJEmFGbolSZKkwgzdkiRJUmH/H//WSC2jim2rAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt = metrics_dataframe.plot(kind='line', figsize=(12,5), x='timestamp', y='value', style='b.', legend=False)\n",
    "plt.set_ylabel(metric_name);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the number of epochs increased, RMSE goes down which is a good sign that the predicted values are getting closer to the actual values from the test set. We can increase number of epochs or change the hyperparameters and try to tweak the model. Let's try to deploy this model and make predictions to see how close the predictions are. Then we can run a hyper-parameter tuning job to determine the best model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utility Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define some common utility functions here that will be used during inference and evaluating results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_protobuf(X, Y=None):\n",
    "    buf = io.BytesIO()\n",
    "    smac.write_spmatrix_to_sparse_tensor(buf, X, Y)\n",
    "    buf.seek(0)\n",
    "    return buf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_sparse_matrix_X(df, nb_rows, nb_customer, nb_products):\n",
    "    # dataframe to array\n",
    "    df_val = df.values\n",
    "\n",
    "    # determine feature size\n",
    "    nb_cols = nb_customer + nb_products\n",
    "    \n",
    "    # extract customers and ratings\n",
    "    df_X = df_val[:,0:2]\n",
    "    # Features are one-hot encoded in a sparse matrix\n",
    "    X = lil_matrix((nb_rows, nb_cols)).astype('float32')\n",
    "    df_X[:,1] = nb_customer + df_X[:,1]\n",
    "    coords = df_X[:,0:2]\n",
    "    X[np.arange(nb_rows), coords[:, 0]] = 1\n",
    "    X[np.arange(nb_rows), coords[:, 1]] = 1\n",
    "\n",
    "    # validate size and shape\n",
    "    assert X.shape == (nb_rows, nb_cols)\n",
    "    \n",
    "    return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real-Time Inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the model is trained, all it takes to deploy the model is a Sagemaker API call `deploy()` that creates the model package, sets up endpoint configuration and finally creates the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------!"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "fm_predictor = fm.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predictions could be done by sending HTTP POST requests from a separate web service, but to keep things easy, we'll just use the `.predict()` method from the SageMaker Python SDK. The API expects JSON or RecordIO format for  request and JSON for response data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_predictor.content_type = 'application/x-recordio-protobuf'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test the model with sample ratings from test data set using `predict()` API call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pb = convert_to_protobuf(test_X[1000:1010]).getvalue()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"predictions\": [{\"score\": 3.8316574096679688}, {\"score\": 3.7261946201324463}, {\"score\": 3.6552248001098633}, {\"score\": 4.027756214141846}, {\"score\": 3.8856489658355713}, {\"score\": 3.7821013927459717}, {\"score\": 3.425412178039551}, {\"score\": 4.217108249664307}, {\"score\": 3.693544387817383}, {\"score\": 4.35484504699707}]}'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = fm_predictor.predict(test_pb)\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3.83, 3.73, 3.66, 4.03, 3.89, 3.78, 3.43, 4.22, 3.69, 4.35]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted = [round(r['score'], 2) for r in json.loads(response)['predictions']]\n",
    "predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual_rating</th>\n",
       "      <th>predicted_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.0</td>\n",
       "      <td>3.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   actual_rating  predicted_rating\n",
       "0            5.0              3.83\n",
       "1            5.0              3.73\n",
       "2            2.0              3.66\n",
       "3            5.0              4.03\n",
       "4            4.0              3.89\n",
       "5            4.0              3.78\n",
       "6            3.0              3.43\n",
       "7            5.0              4.22\n",
       "8            1.0              3.69\n",
       "9            5.0              4.35"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = pd.DataFrame(zip(test_Y[1000:1010], predicted), columns = [\"actual_rating\", \"predicted_rating\"])\n",
    "results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will perform batch inference on the test data set prepared earlier (chunking into multiple protobuf files). To run batch transform, create a model package for the transform endpoint "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Create the model from the training estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_model = fm.create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Perform batch inference on the test data set and save results to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_transformer = fm_model.transformer(\n",
    "    instance_type='ml.c4.xlarge', \n",
    "    instance_count=1, \n",
    "    strategy=\"MultiRecord\", \n",
    "    output_path=\"s3://{}/transform/\".format(bucket)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_transformer.transform(\n",
    "    data=\"s3://{}/prepare/test/\".format(bucket), \n",
    "    data_type='S3Prefix', \n",
    "    content_type=\"application/x-recordio-protobuf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for transform job: factorization-machines-2020-10-27-00-59-2020-10-27-00-59-00-681\n",
      "................................\u001b[34mDocker entrypoint called with argument(s): serve\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.\n",
      "  from numpy.testing import nosetester\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded entry point class algorithm.serve.server_config:config_api\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loading entry points\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/ai_algorithms_sdk/serve.py:195: DeprecationWarning: entrypoint algorithm.request_iterators is deprecated in favor of algorithm.io.data_handlers.serve\n",
      "  \"in favor of algorithm.io.data_handlers.serve\", DeprecationWarning)\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded request iterator application/x-recordio-protobuf\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded request iterator application/json\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded request iterator application/jsonlines\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded response encoder application/x-recordio-protobuf\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded response encoder application/json\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded response encoder application/jsonlines\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loaded entry point class algorithm:model\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] Model Size: 369615766 bytes\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] Number of vCPUs: 4\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] Number of server workers: 4\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [1] [INFO] Starting gunicorn 19.7.1\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [1] [INFO] Using worker: sync\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [35] [INFO] Booting worker with pid: 35\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loading model...\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [36] [INFO] Booting worker with pid: 36\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loading model...\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [37] [INFO] Booting worker with pid: 37\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loading model...\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:05 +0000] [38] [INFO] Booting worker with pid: 38\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:05 INFO 140106874853184] loading model...\u001b[0m\n",
      "\n",
      "\u001b[34m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252709388733 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252799987793 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 WARNING 140106874853184] Requesting context without setting the requested num of gpus. Using 'auto'\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252709388733 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252799987793 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252089500427 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252788066864 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252089500427 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] nvidia-smi took: 0.0252788066864 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:20 INFO 140106874853184] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Model Size: 369615766 bytes\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Number of vCPUs: 4\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"execution_parameters.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760662.149428, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.12956}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Model Size: 369615766 bytes\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Number of vCPUs: 4\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"execution_parameters.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760662.149428, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.12956}\n",
      "\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[34m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:23.141] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpxch5ZC/tmpumksSJ\", \"epoch\": 0, \"duration\": 866, \"num_examples\": 1, \"num_bytes\": 667236}\u001b[0m\n",
      "\u001b[35m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[35m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[35m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] Loading regression model.\u001b[0m\n",
      "\u001b[35m[10/27/2020 01:04:22 INFO 140106874853184] ...model loaded.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:23.141] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpxch5ZC/tmpumksSJ\", \"epoch\": 0, \"duration\": 866, \"num_examples\": 1, \"num_bytes\": 667236}\u001b[0m\n",
      "\u001b[32m2020-10-27T01:04:22.175:[sagemaker logs]: MaxConcurrentTransforms=3, MaxPayloadInMB=6, BatchStrategy=MULTI_RECORD\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:23.539] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmppX_Ygo/tmp1NCiMw\", \"epoch\": 0, \"duration\": 1271, \"num_examples\": 1, \"num_bytes\": 667516}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.602707, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.149826}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:23.626] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp177rB6/tmpIf_HoF\", \"epoch\": 0, \"duration\": 1364, \"num_examples\": 1, \"num_bytes\": 667648}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.678914, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.156319}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.709997, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.192355}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:23.875] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:23.877] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:23.539] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmppX_Ygo/tmp1NCiMw\", \"epoch\": 0, \"duration\": 1271, \"num_examples\": 1, \"num_bytes\": 667516}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.602707, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.149826}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:23.626] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp177rB6/tmpIf_HoF\", \"epoch\": 0, \"duration\": 1364, \"num_examples\": 1, \"num_bytes\": 667648}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.678914, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.156319}\n",
      "\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760663.709997, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.192355}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:23.875] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:23.877] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m/opt/amazon/lib/python2.7/site-packages/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. [u'out_label'])\n",
      "  warnings.warn(msg)\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:24.284] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpIie_Pj/tmpPs6Yqk\", \"epoch\": 0, \"duration\": 408, \"num_examples\": 1, \"num_bytes\": 667248}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:24.284] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpIie_Pj/tmpPs6Yqk\", \"epoch\": 0, \"duration\": 408, \"num_examples\": 1, \"num_bytes\": 667248}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760664.406516, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.710094}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:24.413] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpW12uz9/tmpcVeBIW\", \"epoch\": 0, \"duration\": 536, \"num_examples\": 1, \"num_bytes\": 667688}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:24.470] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760664.510233, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.679023}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:24.545] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:24.946] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpaMFXul/tmph6EWN9\", \"epoch\": 0, \"duration\": 475, \"num_examples\": 1, \"num_bytes\": 667396}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.050605, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.603436}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.126] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.169] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp0QrDua/tmpR0xkPw\", \"epoch\": 0, \"duration\": 623, \"num_examples\": 1, \"num_bytes\": 667556}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.227643, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760664.407049}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.262] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.262] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpO3npl3/tmpM4T6uW\", \"epoch\": 0, \"duration\": 1362, \"num_examples\": 1, \"num_bytes\": 667232}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760664.406516, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.710094}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:24.413] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpW12uz9/tmpcVeBIW\", \"epoch\": 0, \"duration\": 536, \"num_examples\": 1, \"num_bytes\": 667688}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:24.470] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760664.510233, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.679023}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:24.545] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:24.946] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpaMFXul/tmph6EWN9\", \"epoch\": 0, \"duration\": 475, \"num_examples\": 1, \"num_bytes\": 667396}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.050605, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760663.603436}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.126] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.169] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp0QrDua/tmpR0xkPw\", \"epoch\": 0, \"duration\": 623, \"num_examples\": 1, \"num_bytes\": 667556}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.227643, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760664.407049}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.262] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.262] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpO3npl3/tmpM4T6uW\", \"epoch\": 0, \"duration\": 1362, \"num_examples\": 1, \"num_bytes\": 667232}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.328393, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.83335}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.374] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.685] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpW2TbFd/tmptE8gVL\", \"epoch\": 0, \"duration\": 558, \"num_examples\": 1, \"num_bytes\": 667256}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.738643, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.051394}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.779] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpLD6cz4/tmpCUAbsD\", \"epoch\": 0, \"duration\": 515, \"num_examples\": 1, \"num_bytes\": 667536}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.792] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.841835, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760664.510315}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.884] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:25.923] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpQqYXf2/tmpwrzTam\", \"epoch\": 0, \"duration\": 548, \"num_examples\": 1, \"num_bytes\": 667204}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.033772, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.328948}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.069] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.328393, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760662.83335}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.374] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.685] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpW2TbFd/tmptE8gVL\", \"epoch\": 0, \"duration\": 558, \"num_examples\": 1, \"num_bytes\": 667256}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.738643, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.051394}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.779] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpLD6cz4/tmpCUAbsD\", \"epoch\": 0, \"duration\": 515, \"num_examples\": 1, \"num_bytes\": 667536}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.792] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760665.841835, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760664.510315}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.884] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:25.923] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpQqYXf2/tmpwrzTam\", \"epoch\": 0, \"duration\": 548, \"num_examples\": 1, \"num_bytes\": 667204}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.033772, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.328948}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.069] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.289] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpG5X7gA/tmpcnWDm7\", \"epoch\": 0, \"duration\": 495, \"num_examples\": 1, \"num_bytes\": 667236}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.342728, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.228108}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.377] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.429] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpSd0W0b/tmp70WKXY\", \"epoch\": 0, \"duration\": 544, \"num_examples\": 1, \"num_bytes\": 667536}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.489884, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.842006}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.614] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpGILTfx/tmpdopKvS\", \"epoch\": 0, \"duration\": 545, \"num_examples\": 1, \"num_bytes\": 667728}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.622] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.676805, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.738744}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.708] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.810] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpmQ4g88/tmpbq2v7P\", \"epoch\": 0, \"duration\": 432, \"num_examples\": 1, \"num_bytes\": 667220}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.865932, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.034332}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:26.900] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.139] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpZ4rN6c/tmp4tPZiX\", \"epoch\": 0, \"duration\": 516, \"num_examples\": 1, \"num_bytes\": 667588}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.193073, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.343223}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.235] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.289] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpG5X7gA/tmpcnWDm7\", \"epoch\": 0, \"duration\": 495, \"num_examples\": 1, \"num_bytes\": 667236}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.342728, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.228108}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.377] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.429] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpSd0W0b/tmp70WKXY\", \"epoch\": 0, \"duration\": 544, \"num_examples\": 1, \"num_bytes\": 667536}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.489884, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.842006}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.614] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpGILTfx/tmpdopKvS\", \"epoch\": 0, \"duration\": 545, \"num_examples\": 1, \"num_bytes\": 667728}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.622] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.676805, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760665.738744}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.708] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.810] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpmQ4g88/tmpbq2v7P\", \"epoch\": 0, \"duration\": 432, \"num_examples\": 1, \"num_bytes\": 667220}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760666.865932, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.034332}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:26.900] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.139] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpZ4rN6c/tmp4tPZiX\", \"epoch\": 0, \"duration\": 516, \"num_examples\": 1, \"num_bytes\": 667588}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.193073, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.343223}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.235] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.364] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpCn5skS/tmpQqekkA\", \"epoch\": 0, \"duration\": 464, \"num_examples\": 1, \"num_bytes\": 667468}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.425749, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.490372}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.674] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpIaXIuN/tmpcNuU8A\", \"epoch\": 0, \"duration\": 438, \"num_examples\": 1, \"num_bytes\": 667652}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.689] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpuI0cmX/tmpxdrekz\", \"epoch\": 0, \"duration\": 980, \"num_examples\": 1, \"num_bytes\": 252528}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.699] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.70197, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.677211}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.728] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.736226, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.193762}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:27.786] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.194] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp0wNdzE/tmpguytAx\", \"epoch\": 0, \"duration\": 494, \"num_examples\": 1, \"num_bytes\": 667432}\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.249] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpTPaINr/tmphRE5X3\", \"epoch\": 0, \"duration\": 520, \"num_examples\": 1, \"num_bytes\": 667620}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.256363, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.866428}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.364] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpCn5skS/tmpQqekkA\", \"epoch\": 0, \"duration\": 464, \"num_examples\": 1, \"num_bytes\": 667468}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.425749, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.490372}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.674] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpIaXIuN/tmpcNuU8A\", \"epoch\": 0, \"duration\": 438, \"num_examples\": 1, \"num_bytes\": 667652}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.689] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpuI0cmX/tmpxdrekz\", \"epoch\": 0, \"duration\": 980, \"num_examples\": 1, \"num_bytes\": 252528}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.699] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.70197, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.677211}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.728] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760667.736226, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.193762}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:27.786] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.194] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmp0wNdzE/tmpguytAx\", \"epoch\": 0, \"duration\": 494, \"num_examples\": 1, \"num_bytes\": 667432}\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.249] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpTPaINr/tmphRE5X3\", \"epoch\": 0, \"duration\": 520, \"num_examples\": 1, \"num_bytes\": 667620}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.256363, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760666.866428}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.283] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.306995, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.426298}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.329] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.799] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpGBNOoL/tmphRwDFt\", \"epoch\": 0, \"duration\": 516, \"num_examples\": 1, \"num_bytes\": 667496}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.859825, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760668.256458}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.883] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmptBdhQM/tmpiTfa_P\", \"epoch\": 0, \"duration\": 553, \"num_examples\": 1, \"num_bytes\": 667352}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.935236, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760668.307091}\n",
      "\u001b[0m\n",
      "\u001b[34m[2020-10-27 01:04:28.955] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpVc5kd1/tmp8bk9Je\", \"epoch\": 0, \"duration\": 1167, \"num_examples\": 1, \"num_bytes\": 667792}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.990456, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.702023}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.283] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.306995, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.426298}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.329] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.799] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpGBNOoL/tmphRwDFt\", \"epoch\": 0, \"duration\": 516, \"num_examples\": 1, \"num_bytes\": 667496}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.859825, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760668.256458}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.883] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmptBdhQM/tmpiTfa_P\", \"epoch\": 0, \"duration\": 553, \"num_examples\": 1, \"num_bytes\": 667352}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.935236, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760668.307091}\n",
      "\u001b[0m\n",
      "\u001b[35m[2020-10-27 01:04:28.955] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/tmp/tmpVc5kd1/tmp8bk9Je\", \"epoch\": 0, \"duration\": 1167, \"num_examples\": 1, \"num_bytes\": 667792}\u001b[0m\n",
      "\u001b[35m#metrics {\"Metrics\": {\"invocations.count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603760668.990456, \"Dimensions\": {\"Host\": \"UNKNOWN\", \"Operation\": \"scoring\", \"Algorithm\": \"AlgorithmModel\"}, \"StartTime\": 1603760667.702023}\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "print('Waiting for transform job: ' + fm_transformer.latest_transform_job.job_name)\n",
    "fm_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Inference results will be stored in a separate file for each test file chunk. Let's download the results from S3 and merge them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_from_s3(bucket, key):\n",
    "    s3 = boto3.resource('s3')\n",
    "    obj = s3.Object( bucket, key)\n",
    "    content = obj.get()['Body'].read()\n",
    "    return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = []\n",
    "for i in range(N):\n",
    "    key = 'transform/test_' + str(i) + '.protobuf.out'\n",
    "    response = download_from_s3(bucket, key)\n",
    "    result = [json.loads(row)[\"score\"] for row in response.split(\"\\n\") if len(row) > 0]\n",
    "    test_preds.extend(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = np.array(test_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(233781,)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Model Performance\n",
    "\n",
    "Let's start by calculating a naive baseline to approximate how well our model is doing.  The simplest estimate would be to assume every user item rating is just the average rating over all ratings.\n",
    "\n",
    "*Note, we could do better by using each individual video's average, however, in this case it doesn't really matter as the same conclusions would hold.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Naive MSE:', 1.447061540076283)\n"
     ]
    }
   ],
   "source": [
    "print('Naive MSE:', np.mean((test_df['star_rating'] - np.mean(train_df['star_rating'])) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll calculate predictions for our test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('MSE:', 1.1478783285179637)\n"
     ]
    }
   ],
   "source": [
    "print('MSE:', np.mean((test_Y - test_preds) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our neural network and embedding model produces substantially better results (~1.44 vs 1.13 on the mean square error).\n",
    "\n",
    "For recommender systems, subjective accuracy also matters.  Let's get some recommendations for a random user to see if they make intuitive sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>product_id</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>product_title</th>\n",
       "      <th>customer</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55325</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00DAHSY58</td>\n",
       "      <td>5</td>\n",
       "      <td>Under The Dome, Season 1</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96037</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00LI0VA4Q</td>\n",
       "      <td>5</td>\n",
       "      <td>Extant, Season 1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39812</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00L86ZKAK</td>\n",
       "      <td>5</td>\n",
       "      <td>Under The Dome, Season 2</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182878</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B006IX92XO</td>\n",
       "      <td>5</td>\n",
       "      <td>Rome Season 1</td>\n",
       "      <td>6</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83326</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00821OX98</td>\n",
       "      <td>5</td>\n",
       "      <td>Falling Skies Season 1</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160154</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B008BQG3RE</td>\n",
       "      <td>5</td>\n",
       "      <td>Falling Skies Season 2</td>\n",
       "      <td>6</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227942</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00I8H6MGS</td>\n",
       "      <td>5</td>\n",
       "      <td>Dallas Buyers Club</td>\n",
       "      <td>6</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>569635</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00F35B8F8</td>\n",
       "      <td>5</td>\n",
       "      <td>Star Trek Into Darkness (Plus Bonus Features)</td>\n",
       "      <td>6</td>\n",
       "      <td>216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>474425</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B004HWYTP4</td>\n",
       "      <td>5</td>\n",
       "      <td>Red</td>\n",
       "      <td>6</td>\n",
       "      <td>226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161472</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B0077RUCRK</td>\n",
       "      <td>5</td>\n",
       "      <td>Carnivale: Season 2</td>\n",
       "      <td>6</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>895607</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B001BR5NLW</td>\n",
       "      <td>1</td>\n",
       "      <td>The Chronicles of Riddick - Unrated Director's...</td>\n",
       "      <td>6</td>\n",
       "      <td>6352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876748</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B005BZQS1E</td>\n",
       "      <td>1</td>\n",
       "      <td>Your Highness</td>\n",
       "      <td>6</td>\n",
       "      <td>6977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1020854</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B004GWL3FE</td>\n",
       "      <td>1</td>\n",
       "      <td>Dinner for Schmucks</td>\n",
       "      <td>6</td>\n",
       "      <td>7712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1032119</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00SY9R5FC</td>\n",
       "      <td>1</td>\n",
       "      <td>Suburban Gothic</td>\n",
       "      <td>6</td>\n",
       "      <td>10243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>881306</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B000I9U8MI</td>\n",
       "      <td>1</td>\n",
       "      <td>Never Been Kissed</td>\n",
       "      <td>6</td>\n",
       "      <td>10438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1060730</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00TY68T6I</td>\n",
       "      <td>1</td>\n",
       "      <td>The Returned Season 1</td>\n",
       "      <td>6</td>\n",
       "      <td>11403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938151</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00KQFOLBI</td>\n",
       "      <td>1</td>\n",
       "      <td>Mirage Men</td>\n",
       "      <td>6</td>\n",
       "      <td>13011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>957084</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B001JDQXCG</td>\n",
       "      <td>1</td>\n",
       "      <td>The New World (2005)</td>\n",
       "      <td>6</td>\n",
       "      <td>13576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>900955</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00IAKN6V2</td>\n",
       "      <td>1</td>\n",
       "      <td>Welcome To The Jungle</td>\n",
       "      <td>6</td>\n",
       "      <td>17271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996252</th>\n",
       "      <td>50818682</td>\n",
       "      <td>B00A4OC0LO</td>\n",
       "      <td>1</td>\n",
       "      <td>The Love You Save</td>\n",
       "      <td>6</td>\n",
       "      <td>20128</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         customer_id  product_id  star_rating  \\\n",
       "55325       50818682  B00DAHSY58            5   \n",
       "96037       50818682  B00LI0VA4Q            5   \n",
       "39812       50818682  B00L86ZKAK            5   \n",
       "182878      50818682  B006IX92XO            5   \n",
       "83326       50818682  B00821OX98            5   \n",
       "160154      50818682  B008BQG3RE            5   \n",
       "227942      50818682  B00I8H6MGS            5   \n",
       "569635      50818682  B00F35B8F8            5   \n",
       "474425      50818682  B004HWYTP4            5   \n",
       "161472      50818682  B0077RUCRK            5   \n",
       "895607      50818682  B001BR5NLW            1   \n",
       "876748      50818682  B005BZQS1E            1   \n",
       "1020854     50818682  B004GWL3FE            1   \n",
       "1032119     50818682  B00SY9R5FC            1   \n",
       "881306      50818682  B000I9U8MI            1   \n",
       "1060730     50818682  B00TY68T6I            1   \n",
       "938151      50818682  B00KQFOLBI            1   \n",
       "957084      50818682  B001JDQXCG            1   \n",
       "900955      50818682  B00IAKN6V2            1   \n",
       "996252      50818682  B00A4OC0LO            1   \n",
       "\n",
       "                                             product_title  customer  product  \n",
       "55325                             Under The Dome, Season 1         6        5  \n",
       "96037                                     Extant, Season 1         6        6  \n",
       "39812                             Under The Dome, Season 2         6       21  \n",
       "182878                                       Rome Season 1         6       27  \n",
       "83326                               Falling Skies Season 1         6       40  \n",
       "160154                              Falling Skies Season 2         6       80  \n",
       "227942                                  Dallas Buyers Club         6       98  \n",
       "569635       Star Trek Into Darkness (Plus Bonus Features)         6      216  \n",
       "474425                                                 Red         6      226  \n",
       "161472                                 Carnivale: Season 2         6      286  \n",
       "895607   The Chronicles of Riddick - Unrated Director's...         6     6352  \n",
       "876748                                       Your Highness         6     6977  \n",
       "1020854                                Dinner for Schmucks         6     7712  \n",
       "1032119                                    Suburban Gothic         6    10243  \n",
       "881306                                   Never Been Kissed         6    10438  \n",
       "1060730                              The Returned Season 1         6    11403  \n",
       "938151                                          Mirage Men         6    13011  \n",
       "957084                                The New World (2005)         6    13576  \n",
       "900955                               Welcome To The Jungle         6    17271  \n",
       "996252                                   The Love You Save         6    20128  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_customer_6 = reduced_df[reduced_df['customer'] == 6].sort_values(['star_rating', 'product'], ascending=[False, True])\n",
    "pd.concat((df_customer_6.head(10), df_customer_6.tail(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, user #6 seems to like sprawling dramamtic television series and sci-fi, but they dislike silly comedies.\n",
    "\n",
    "Now we'll loop through and predict user #6's ratings for every common video in the catalog, to see which ones we'd recommend and which ones we wouldn't."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_payload(cust_id, nb_customer, nb_products, product_index):\n",
    "    # prepare payload for user #6\n",
    "    c = [cust_id] * nb_products\n",
    "    p = product_index['product'].values\n",
    "    x = pd.DataFrame(zip(c,p))\n",
    "    p_x = convert_sparse_matrix_X(x, x.shape[0], nb_customer, nb_products)\n",
    "    x_pb = convert_to_protobuf(p_x)\n",
    "    return x_pb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_pb = create_payload(6, nb_customer, nb_products, product_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make predictions using end-point created in Real-Time Inference\n",
    "response = fm_predictor.predict(x_pb)\n",
    "predictions = [round(r['score'], 2) for r in json.loads(response)['predictions']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_df = pd.DataFrame({'product': product_index['product'],\n",
    "                            'prediction': predictions})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>product</th>\n",
       "      <th>product_id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>40</td>\n",
       "      <td>B00821OX98</td>\n",
       "      <td>Falling Skies Season 1</td>\n",
       "      <td>5</td>\n",
       "      <td>5.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>216</td>\n",
       "      <td>B00F35B8F8</td>\n",
       "      <td>Star Trek Into Darkness (Plus Bonus Features)</td>\n",
       "      <td>5</td>\n",
       "      <td>5.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1710</td>\n",
       "      <td>B009CP81EQ</td>\n",
       "      <td>Bones Season 8</td>\n",
       "      <td>5</td>\n",
       "      <td>5.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>911</td>\n",
       "      <td>B002BI6SMI</td>\n",
       "      <td>Taken</td>\n",
       "      <td>5</td>\n",
       "      <td>5.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>5</td>\n",
       "      <td>B00DAHSY58</td>\n",
       "      <td>Under The Dome, Season 1</td>\n",
       "      <td>5</td>\n",
       "      <td>5.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>226</td>\n",
       "      <td>B004HWYTP4</td>\n",
       "      <td>Red</td>\n",
       "      <td>5</td>\n",
       "      <td>5.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>716</td>\n",
       "      <td>B00F406S2U</td>\n",
       "      <td>Arrow: The Complete Second Season</td>\n",
       "      <td>5</td>\n",
       "      <td>5.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1499</td>\n",
       "      <td>B00F49F96C</td>\n",
       "      <td>Almost Human: The Complete First Season</td>\n",
       "      <td>5</td>\n",
       "      <td>5.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>978</td>\n",
       "      <td>B00NEFWXNK</td>\n",
       "      <td>Arrow: Season 3</td>\n",
       "      <td>5</td>\n",
       "      <td>5.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2285</td>\n",
       "      <td>B001LLGA7E</td>\n",
       "      <td>Heroes Volume 1</td>\n",
       "      <td>5</td>\n",
       "      <td>5.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2170</td>\n",
       "      <td>B00F4AST04</td>\n",
       "      <td>The Heat</td>\n",
       "      <td>5</td>\n",
       "      <td>5.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>3911</td>\n",
       "      <td>B001O31VVE</td>\n",
       "      <td>The Wedding Date</td>\n",
       "      <td>5</td>\n",
       "      <td>5.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>80</td>\n",
       "      <td>B008BQG3RE</td>\n",
       "      <td>Falling Skies Season 2</td>\n",
       "      <td>5</td>\n",
       "      <td>5.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2656</td>\n",
       "      <td>B00B0TNZ02</td>\n",
       "      <td>The Twilight Saga: Breaking Dawn Part 2</td>\n",
       "      <td>5</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>6946</td>\n",
       "      <td>B0094M2W24</td>\n",
       "      <td>Broken Trail</td>\n",
       "      <td>5</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>432</td>\n",
       "      <td>B00KKGS1KK</td>\n",
       "      <td>Falling Skies Season 4</td>\n",
       "      <td>5</td>\n",
       "      <td>4.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1956</td>\n",
       "      <td>B000NGTJTY</td>\n",
       "      <td>The Departed</td>\n",
       "      <td>5</td>\n",
       "      <td>4.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1304</td>\n",
       "      <td>B004IYDSPS</td>\n",
       "      <td>Life As We Know It (2010)</td>\n",
       "      <td>5</td>\n",
       "      <td>4.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>8924</td>\n",
       "      <td>B00KCGSRYI</td>\n",
       "      <td>Enemy Of The State</td>\n",
       "      <td>5</td>\n",
       "      <td>4.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2678</td>\n",
       "      <td>B003XQM2YU</td>\n",
       "      <td>Robin Hood (Unrated)</td>\n",
       "      <td>5</td>\n",
       "      <td>4.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>4670</td>\n",
       "      <td>B0026O475M</td>\n",
       "      <td>Cast Away</td>\n",
       "      <td>5</td>\n",
       "      <td>4.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1058</td>\n",
       "      <td>B009329Y30</td>\n",
       "      <td>The Lucky One</td>\n",
       "      <td>5</td>\n",
       "      <td>4.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>98</td>\n",
       "      <td>B00I8H6MGS</td>\n",
       "      <td>Dallas Buyers Club</td>\n",
       "      <td>5</td>\n",
       "      <td>4.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>3528</td>\n",
       "      <td>B009NXVGX0</td>\n",
       "      <td>Eye for an Eye</td>\n",
       "      <td>5</td>\n",
       "      <td>4.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>1352</td>\n",
       "      <td>B001M432XA</td>\n",
       "      <td>It's A Wonderful Life (Black &amp; White Version)</td>\n",
       "      <td>5</td>\n",
       "      <td>4.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2766</td>\n",
       "      <td>B001GY1JDQ</td>\n",
       "      <td>Man On Fire</td>\n",
       "      <td>5</td>\n",
       "      <td>4.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>6022</td>\n",
       "      <td>B0063B2BNE</td>\n",
       "      <td>Bones Season 7</td>\n",
       "      <td>5</td>\n",
       "      <td>4.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>8371</td>\n",
       "      <td>B000JZ3TYA</td>\n",
       "      <td>The Game</td>\n",
       "      <td>5</td>\n",
       "      <td>4.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>11868</td>\n",
       "      <td>B000W5KBI4</td>\n",
       "      <td>Numb3rs Season 4</td>\n",
       "      <td>5</td>\n",
       "      <td>4.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>15367</td>\n",
       "      <td>B000IK882Y</td>\n",
       "      <td>Bones Season 2</td>\n",
       "      <td>5</td>\n",
       "      <td>4.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>13011</td>\n",
       "      <td>B00KQFOLBI</td>\n",
       "      <td>Mirage Men</td>\n",
       "      <td>1</td>\n",
       "      <td>3.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>3775</td>\n",
       "      <td>B00F49AV2E</td>\n",
       "      <td>The Tomorrow People: The Complete First Season</td>\n",
       "      <td>3</td>\n",
       "      <td>3.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2635</td>\n",
       "      <td>B00D5P20UA</td>\n",
       "      <td>Sinbad Season 1</td>\n",
       "      <td>3</td>\n",
       "      <td>3.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>10213</td>\n",
       "      <td>B009ENNENE</td>\n",
       "      <td>You May Not Kiss The Bride</td>\n",
       "      <td>3</td>\n",
       "      <td>3.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>11403</td>\n",
       "      <td>B00TY68T6I</td>\n",
       "      <td>The Returned Season 1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>6951</td>\n",
       "      <td>B008EQIA7W</td>\n",
       "      <td>Wrath of the Titans</td>\n",
       "      <td>3</td>\n",
       "      <td>3.36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>222</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>647</td>\n",
       "      <td>B00JB3NUHM</td>\n",
       "      <td>Noah</td>\n",
       "      <td>4</td>\n",
       "      <td>3.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>25473</td>\n",
       "      <td>B001EYVT74</td>\n",
       "      <td>Code 46</td>\n",
       "      <td>2</td>\n",
       "      <td>3.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>4512</td>\n",
       "      <td>B00HQCSFTI</td>\n",
       "      <td>Murph: The Protector</td>\n",
       "      <td>2</td>\n",
       "      <td>3.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>312</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>5208</td>\n",
       "      <td>B004EQJHLE</td>\n",
       "      <td>Bread &amp; Tulips (English Subtitled)</td>\n",
       "      <td>1</td>\n",
       "      <td>3.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>5643</td>\n",
       "      <td>B00KD9K6KC</td>\n",
       "      <td>A Night in Old Mexico</td>\n",
       "      <td>3</td>\n",
       "      <td>3.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>18885</td>\n",
       "      <td>B00J2PQPBC</td>\n",
       "      <td>Sparks</td>\n",
       "      <td>2</td>\n",
       "      <td>3.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>907</td>\n",
       "      <td>B00OCEDVW4</td>\n",
       "      <td>Automata</td>\n",
       "      <td>5</td>\n",
       "      <td>3.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>277</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>4341</td>\n",
       "      <td>B00BT0ZDF2</td>\n",
       "      <td>Cloud Atlas +Bonus Features</td>\n",
       "      <td>3</td>\n",
       "      <td>3.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>13576</td>\n",
       "      <td>B001JDQXCG</td>\n",
       "      <td>The New World (2005)</td>\n",
       "      <td>1</td>\n",
       "      <td>3.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>17271</td>\n",
       "      <td>B00IAKN6V2</td>\n",
       "      <td>Welcome To The Jungle</td>\n",
       "      <td>1</td>\n",
       "      <td>3.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>10438</td>\n",
       "      <td>B000I9U8MI</td>\n",
       "      <td>Never Been Kissed</td>\n",
       "      <td>1</td>\n",
       "      <td>3.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>5891</td>\n",
       "      <td>B005MSD75U</td>\n",
       "      <td>Zookeeper</td>\n",
       "      <td>3</td>\n",
       "      <td>3.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>7811</td>\n",
       "      <td>B001HQKB1Y</td>\n",
       "      <td>Baby Mama</td>\n",
       "      <td>2</td>\n",
       "      <td>3.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>3041</td>\n",
       "      <td>B0088WG89O</td>\n",
       "      <td>Journey 2: The Mysterious Island</td>\n",
       "      <td>3</td>\n",
       "      <td>3.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>2580</td>\n",
       "      <td>B00O4U8WNE</td>\n",
       "      <td>Horns</td>\n",
       "      <td>3</td>\n",
       "      <td>3.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>5530</td>\n",
       "      <td>B001BXQ95U</td>\n",
       "      <td>The Flash Season 1 (Classic Series)</td>\n",
       "      <td>2</td>\n",
       "      <td>3.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>7712</td>\n",
       "      <td>B004GWL3FE</td>\n",
       "      <td>Dinner for Schmucks</td>\n",
       "      <td>1</td>\n",
       "      <td>3.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>267</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>738</td>\n",
       "      <td>B00HUNO3JE</td>\n",
       "      <td>Enough Said</td>\n",
       "      <td>3</td>\n",
       "      <td>3.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>8004</td>\n",
       "      <td>B008FWHUQM</td>\n",
       "      <td>360</td>\n",
       "      <td>2</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>851</td>\n",
       "      <td>B00G32Q4CC</td>\n",
       "      <td>The Internship</td>\n",
       "      <td>1</td>\n",
       "      <td>2.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>516</td>\n",
       "      <td>B00NTSYP3S</td>\n",
       "      <td>Neighbors</td>\n",
       "      <td>3</td>\n",
       "      <td>2.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>465</td>\n",
       "      <td>B009329ZF2</td>\n",
       "      <td>Battleship</td>\n",
       "      <td>1</td>\n",
       "      <td>2.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>30</td>\n",
       "      <td>B006MYGL8S</td>\n",
       "      <td>Deadwood Season 1</td>\n",
       "      <td>3</td>\n",
       "      <td>2.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>6</td>\n",
       "      <td>50818682</td>\n",
       "      <td>575</td>\n",
       "      <td>B00HYTNBU6</td>\n",
       "      <td>All Is Lost</td>\n",
       "      <td>2</td>\n",
       "      <td>2.27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>323 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     customer  customer_id  product  product_id  \\\n",
       "4           6     50818682       40  B00821OX98   \n",
       "7           6     50818682      216  B00F35B8F8   \n",
       "43          6     50818682     1710  B009CP81EQ   \n",
       "26          6     50818682      911  B002BI6SMI   \n",
       "0           6     50818682        5  B00DAHSY58   \n",
       "8           6     50818682      226  B004HWYTP4   \n",
       "23          6     50818682      716  B00F406S2U   \n",
       "38          6     50818682     1499  B00F49F96C   \n",
       "27          6     50818682      978  B00NEFWXNK   \n",
       "52          6     50818682     2285  B001LLGA7E   \n",
       "48          6     50818682     2170  B00F4AST04   \n",
       "83          6     50818682     3911  B001O31VVE   \n",
       "5           6     50818682       80  B008BQG3RE   \n",
       "59          6     50818682     2656  B00B0TNZ02   \n",
       "129         6     50818682     6946  B0094M2W24   \n",
       "15          6     50818682      432  B00KKGS1KK   \n",
       "45          6     50818682     1956  B000NGTJTY   \n",
       "33          6     50818682     1304  B004IYDSPS   \n",
       "155         6     50818682     8924  B00KCGSRYI   \n",
       "60          6     50818682     2678  B003XQM2YU   \n",
       "94          6     50818682     4670  B0026O475M   \n",
       "29          6     50818682     1058  B009329Y30   \n",
       "6           6     50818682       98  B00I8H6MGS   \n",
       "76          6     50818682     3528  B009NXVGX0   \n",
       "35          6     50818682     1352  B001M432XA   \n",
       "62          6     50818682     2766  B001GY1JDQ   \n",
       "116         6     50818682     6022  B0063B2BNE   \n",
       "144         6     50818682     8371  B000JZ3TYA   \n",
       "165         6     50818682    11868  B000W5KBI4   \n",
       "185         6     50818682    15367  B000IK882Y   \n",
       "..        ...          ...      ...         ...   \n",
       "319         6     50818682    13011  B00KQFOLBI   \n",
       "275         6     50818682     3775  B00F49AV2E   \n",
       "273         6     50818682     2635  B00D5P20UA   \n",
       "284         6     50818682    10213  B009ENNENE   \n",
       "318         6     50818682    11403  B00TY68T6I   \n",
       "283         6     50818682     6951  B008EQIA7W   \n",
       "222         6     50818682      647  B00JB3NUHM   \n",
       "304         6     50818682    25473  B001EYVT74   \n",
       "296         6     50818682     4512  B00HQCSFTI   \n",
       "312         6     50818682     5208  B004EQJHLE   \n",
       "281         6     50818682     5643  B00KD9K6KC   \n",
       "303         6     50818682    18885  B00J2PQPBC   \n",
       "25          6     50818682      907  B00OCEDVW4   \n",
       "277         6     50818682     4341  B00BT0ZDF2   \n",
       "320         6     50818682    13576  B001JDQXCG   \n",
       "321         6     50818682    17271  B00IAKN6V2   \n",
       "317         6     50818682    10438  B000I9U8MI   \n",
       "282         6     50818682     5891  B005MSD75U   \n",
       "300         6     50818682     7811  B001HQKB1Y   \n",
       "274         6     50818682     3041  B0088WG89O   \n",
       "272         6     50818682     2580  B00O4U8WNE   \n",
       "298         6     50818682     5530  B001BXQ95U   \n",
       "315         6     50818682     7712  B004GWL3FE   \n",
       "267         6     50818682      738  B00HUNO3JE   \n",
       "301         6     50818682     8004  B008FWHUQM   \n",
       "310         6     50818682      851  B00G32Q4CC   \n",
       "266         6     50818682      516  B00NTSYP3S   \n",
       "308         6     50818682      465  B009329ZF2   \n",
       "265         6     50818682       30  B006MYGL8S   \n",
       "293         6     50818682      575  B00HYTNBU6   \n",
       "\n",
       "                                      product_title  star_rating  prediction  \n",
       "4                            Falling Skies Season 1            5        5.49  \n",
       "7     Star Trek Into Darkness (Plus Bonus Features)            5        5.28  \n",
       "43                                   Bones Season 8            5        5.25  \n",
       "26                                            Taken            5        5.22  \n",
       "0                          Under The Dome, Season 1            5        5.13  \n",
       "8                                               Red            5        5.12  \n",
       "23                Arrow: The Complete Second Season            5        5.11  \n",
       "38          Almost Human: The Complete First Season            5        5.11  \n",
       "27                                  Arrow: Season 3            5        5.09  \n",
       "52                                  Heroes Volume 1            5        5.07  \n",
       "48                                         The Heat            5        5.04  \n",
       "83                                 The Wedding Date            5        5.03  \n",
       "5                            Falling Skies Season 2            5        5.02  \n",
       "59          The Twilight Saga: Breaking Dawn Part 2            5        4.98  \n",
       "129                                    Broken Trail            5        4.98  \n",
       "15                           Falling Skies Season 4            5        4.97  \n",
       "45                                     The Departed            5        4.96  \n",
       "33                        Life As We Know It (2010)            5        4.94  \n",
       "155                              Enemy Of The State            5        4.94  \n",
       "60                             Robin Hood (Unrated)            5        4.92  \n",
       "94                                        Cast Away            5        4.92  \n",
       "29                                    The Lucky One            5        4.91  \n",
       "6                                Dallas Buyers Club            5        4.90  \n",
       "76                                   Eye for an Eye            5        4.89  \n",
       "35    It's A Wonderful Life (Black & White Version)            5        4.87  \n",
       "62                                      Man On Fire            5        4.87  \n",
       "116                                  Bones Season 7            5        4.87  \n",
       "144                                        The Game            5        4.87  \n",
       "165                                Numb3rs Season 4            5        4.87  \n",
       "185                                  Bones Season 2            5        4.86  \n",
       "..                                              ...          ...         ...  \n",
       "319                                      Mirage Men            1        3.44  \n",
       "275  The Tomorrow People: The Complete First Season            3        3.40  \n",
       "273                                 Sinbad Season 1            3        3.38  \n",
       "284                      You May Not Kiss The Bride            3        3.37  \n",
       "318                           The Returned Season 1            1        3.37  \n",
       "283                             Wrath of the Titans            3        3.36  \n",
       "222                                            Noah            4        3.35  \n",
       "304                                         Code 46            2        3.29  \n",
       "296                            Murph: The Protector            2        3.28  \n",
       "312              Bread & Tulips (English Subtitled)            1        3.28  \n",
       "281                           A Night in Old Mexico            3        3.28  \n",
       "303                                          Sparks            2        3.28  \n",
       "25                                         Automata            5        3.25  \n",
       "277                     Cloud Atlas +Bonus Features            3        3.25  \n",
       "320                            The New World (2005)            1        3.25  \n",
       "321                           Welcome To The Jungle            1        3.24  \n",
       "317                               Never Been Kissed            1        3.23  \n",
       "282                                       Zookeeper            3        3.22  \n",
       "300                                       Baby Mama            2        3.21  \n",
       "274                Journey 2: The Mysterious Island            3        3.16  \n",
       "272                                           Horns            3        3.13  \n",
       "298             The Flash Season 1 (Classic Series)            2        3.08  \n",
       "315                             Dinner for Schmucks            1        3.06  \n",
       "267                                     Enough Said            3        3.05  \n",
       "301                                             360            2        2.94  \n",
       "310                                  The Internship            1        2.79  \n",
       "266                                       Neighbors            3        2.65  \n",
       "308                                      Battleship            1        2.58  \n",
       "265                               Deadwood Season 1            3        2.38  \n",
       "293                                     All Is Lost            2        2.27  \n",
       "\n",
       "[323 rows x 7 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results_cust_6 = df_customer_6.merge(predictions_df, on=['product'])[['customer', 'customer_id', 'product', 'product_id', 'product_title', 'star_rating', 'prediction']]\n",
    "df_results_cust_6.sort_values(['prediction', 'product'], ascending=[False, True])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, our predicted highly rated shows have some well-reviewed TV dramas and some sci-fi.  Meanwhile, our bottom rated shows include goofball comedies.\n",
    "\n",
    "*Note, because of random initialization in the weights, results on subsequent runs may differ slightly.*\n",
    "\n",
    "Let's confirm that we no longer have almost perfect correlation in recommendations with user #7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_pb = create_payload(7, nb_customer, nb_products, product_index)\n",
    "response = fm_predictor.predict(x_pb)\n",
    "predictions_user7 = [round(r['score'], 2) for r in json.loads(response)['predictions']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'AxesSubplot' object has no attribute 'show'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0mTraceback (most recent call last)",
      "\u001b[0;32m<ipython-input-69-582ba3dbfe69>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpredictions_df\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'prediction'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpredictions_user7\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'AxesSubplot' object has no attribute 'show'"
     ]
    }
   ],
   "source": [
    "plt.scatter(predictions_df['prediction'], np.array(predictions_user7))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Tuning\n",
    "\n",
    "So far, we have developed a deep learning model to predict customer ratings but the model could be improved further by various techniques. In this section, let's see if tuning the hyper-parameters of Factorization Machine is going to make the model any better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_location = 's3://{}/train/'.format(bucket)\n",
    "s3_train_path = 's3://{}/prepare/train/train.protobuf'.format(bucket)\n",
    "s3_val_path = 's3://{}/prepare/validate/validate.protobuf'.format(bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's create the estimator with Factorization Machines container similar to how we defined in training the model. Also, set the initial hyper-parameters that we know worked before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_estimator = sagemaker.estimator.Estimator(container,\n",
    "                                   role, \n",
    "                                   train_instance_count=1, \n",
    "                                   train_instance_type='ml.c5.4xlarge',\n",
    "                                   output_path=output_location,\n",
    "                                   sagemaker_session=sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_estimator.set_hyperparameters(\n",
    "    feature_dim=feature_dim,\n",
    "    predictor_type='regressor',\n",
    "    mini_batch_size=200,\n",
    "    num_factors=512,\n",
    "    bias_lr=0.02,\n",
    "    epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Find best hyperparameters with Sagemaker's Automatic Model Tuning. Following hyperparameters will be tuned\n",
    "    - ***factors_lr:*** The learning rate for factorization terms.\n",
    "    - ***factors_init_sigma:*** The standard deviation for initialization of factorization terms. Takes effect if factors_init_method is set to normal.\n",
    "    \n",
    "\n",
    "- Define the hyperparameter tuning ranges to be searched and set the objective metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameter_ranges=  {\n",
    "    \"factors_lr\": ContinuousParameter(0.0001, 0.2),\n",
    "    \"factors_init_sigma\": ContinuousParameter(0.0001, 1)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now that we have our ranges defined we want to define our success metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = \"test:rmse\"\n",
    "objective_type = \"Minimize\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Start hyperparameter tuning job with the ranges defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_tuner = HyperparameterTuner(\n",
    "    estimator=fm_estimator,\n",
    "    objective_metric_name=objective_metric_name, \n",
    "    hyperparameter_ranges=hyperparameter_ranges,\n",
    "    objective_type=objective_type,\n",
    "    max_jobs=10,\n",
    "    max_parallel_jobs=2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp_prefix = time.strftime(\"%Y%m%d-%H%M%S\", time.gmtime())\n",
    "fm_tuner_job_name = 'hpo-fm-' + timestamp_prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "fm_tuner.fit({'train': s3_train_path, 'test': s3_val_path}, job_name=fm_tuner_job_name, wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Track hyperparameter tuning job progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reminder: the tuning job has not been completed.\n",
      "0 training jobs have completed\n"
     ]
    }
   ],
   "source": [
    "# run this cell to check current status of hyperparameter tuning job\n",
    "tuning_job_result = sm.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=fm_tuner_job_name)\n",
    "\n",
    "status = tuning_job_result['HyperParameterTuningJobStatus']\n",
    "if status != 'Completed':\n",
    "    print('Reminder: the tuning job has not been completed.')\n",
    "    \n",
    "job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']\n",
    "print(\"%d training jobs have completed\" % job_count)\n",
    "    \n",
    "is_minimize = (tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['Type'] != 'Maximize')\n",
    "objective_name = tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['MetricName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\n",
      "CPU times: user 9.42 ms, sys: 7.72 ms, total: 17.1 ms\n",
      "Wall time: 1.13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "fm_tuner.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Analyze Hyper-Parameter Tuning Job Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>factors_init_sigma</th>\n",
       "      <th>factors_lr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.096982</td>\n",
       "      <td>498.0</td>\n",
       "      <td>2020-10-27 05:50:34+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-010-fdf6d9fa</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:42:16+00:00</td>\n",
       "      <td>0.098703</td>\n",
       "      <td>0.000103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.363452</td>\n",
       "      <td>492.0</td>\n",
       "      <td>2020-10-27 05:49:40+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-009-8f38cd6f</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:41:28+00:00</td>\n",
       "      <td>0.192474</td>\n",
       "      <td>0.000107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.103914</td>\n",
       "      <td>473.0</td>\n",
       "      <td>2020-10-27 05:39:05+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-008-0c0dcf5d</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:31:12+00:00</td>\n",
       "      <td>0.001125</td>\n",
       "      <td>0.000147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.073588</td>\n",
       "      <td>499.0</td>\n",
       "      <td>2020-10-27 05:39:18+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-007-4f34d0fd</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:30:59+00:00</td>\n",
       "      <td>0.000137</td>\n",
       "      <td>0.000110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.132511</td>\n",
       "      <td>463.0</td>\n",
       "      <td>2020-10-27 05:29:07+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-006-a39a4ecf</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:21:24+00:00</td>\n",
       "      <td>0.000177</td>\n",
       "      <td>0.000235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.159438</td>\n",
       "      <td>467.0</td>\n",
       "      <td>2020-10-27 05:28:36+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-005-38c8de1e</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:20:49+00:00</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>0.000387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.076811</td>\n",
       "      <td>462.0</td>\n",
       "      <td>2020-10-27 05:16:51+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-004-8da1715c</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:09:09+00:00</td>\n",
       "      <td>0.021933</td>\n",
       "      <td>0.000135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.061027</td>\n",
       "      <td>511.0</td>\n",
       "      <td>2020-10-27 05:16:59+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-003-ba3bbbca</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:08:28+00:00</td>\n",
       "      <td>0.016572</td>\n",
       "      <td>0.000100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.326390</td>\n",
       "      <td>493.0</td>\n",
       "      <td>2020-10-27 05:04:46+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-002-2b0ccbe3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 04:56:33+00:00</td>\n",
       "      <td>0.030746</td>\n",
       "      <td>0.000490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>12.109571</td>\n",
       "      <td>490.0</td>\n",
       "      <td>2020-10-27 05:04:36+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-001-d6d6d635</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 04:56:26+00:00</td>\n",
       "      <td>0.949034</td>\n",
       "      <td>0.044512</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FinalObjectiveValue  TrainingElapsedTimeSeconds           TrainingEndTime  \\\n",
       "0             1.096982                       498.0 2020-10-27 05:50:34+00:00   \n",
       "1             1.363452                       492.0 2020-10-27 05:49:40+00:00   \n",
       "2             1.103914                       473.0 2020-10-27 05:39:05+00:00   \n",
       "3             1.073588                       499.0 2020-10-27 05:39:18+00:00   \n",
       "4             1.132511                       463.0 2020-10-27 05:29:07+00:00   \n",
       "5             1.159438                       467.0 2020-10-27 05:28:36+00:00   \n",
       "6             1.076811                       462.0 2020-10-27 05:16:51+00:00   \n",
       "7             1.061027                       511.0 2020-10-27 05:16:59+00:00   \n",
       "8             1.326390                       493.0 2020-10-27 05:04:46+00:00   \n",
       "9            12.109571                       490.0 2020-10-27 05:04:36+00:00   \n",
       "\n",
       "                       TrainingJobName TrainingJobStatus  \\\n",
       "0  hpo-fm-20201027-045403-010-fdf6d9fa         Completed   \n",
       "1  hpo-fm-20201027-045403-009-8f38cd6f         Completed   \n",
       "2  hpo-fm-20201027-045403-008-0c0dcf5d         Completed   \n",
       "3  hpo-fm-20201027-045403-007-4f34d0fd         Completed   \n",
       "4  hpo-fm-20201027-045403-006-a39a4ecf         Completed   \n",
       "5  hpo-fm-20201027-045403-005-38c8de1e         Completed   \n",
       "6  hpo-fm-20201027-045403-004-8da1715c         Completed   \n",
       "7  hpo-fm-20201027-045403-003-ba3bbbca         Completed   \n",
       "8  hpo-fm-20201027-045403-002-2b0ccbe3         Completed   \n",
       "9  hpo-fm-20201027-045403-001-d6d6d635         Completed   \n",
       "\n",
       "          TrainingStartTime  factors_init_sigma  factors_lr  \n",
       "0 2020-10-27 05:42:16+00:00            0.098703    0.000103  \n",
       "1 2020-10-27 05:41:28+00:00            0.192474    0.000107  \n",
       "2 2020-10-27 05:31:12+00:00            0.001125    0.000147  \n",
       "3 2020-10-27 05:30:59+00:00            0.000137    0.000110  \n",
       "4 2020-10-27 05:21:24+00:00            0.000177    0.000235  \n",
       "5 2020-10-27 05:20:49+00:00            0.000100    0.000387  \n",
       "6 2020-10-27 05:09:09+00:00            0.021933    0.000135  \n",
       "7 2020-10-27 05:08:28+00:00            0.016572    0.000100  \n",
       "8 2020-10-27 04:56:33+00:00            0.030746    0.000490  \n",
       "9 2020-10-27 04:56:26+00:00            0.949034    0.044512  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# plug-in the training job name and metrics to be captured\n",
    "fm_tuner_analytics = HyperparameterTuningJobAnalytics(hyperparameter_tuning_job_name=fm_tuner_job_name)\n",
    "df_fm_tuner_metrics = fm_tuner_analytics.dataframe()\n",
    "df_fm_tuner_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtIAAAE5CAYAAAC9Acd2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi41LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvSM8oowAAHnRJREFUeJzt3XmUZWV97vFvd3VD29BQDCVqiIqIv4CoyKQggld0ObvUgEQRI16iLq8myzhlCYogagS95l414IRGUQFZijMmxgsiHQVEVBx+KypicEAQChrapqe6f+xdWFa66z31dp2hzvl+1urVVaf2qf3W07vPfmqfd++9ZGpqCkmSJEnzs7TfA5AkSZIWI4u0JEmSVMEiLUmSJFWwSEuSJEkVLNKSJElSBYu0JEmSVMEiLUmSJFWwSEuSJEkVLNKSJElSBYu0JEmSVGFZvwfQIe9jLkmSpF5Y0umCi6VIc/PNa/o9hAUzPr6Sycm1/R7GQDOjMjMqM6MyMyozozIzKjOjskHIaGJi1byWd2qHJEmSVMEiLUmSJFWwSEuSJEkVLNKSJElSBYu0JEmSVMEiLUmSJFWwSEuSJEkVFk2R3rBxc7+HIEmSJN1j0RTpU8+90jItSZKkgdG1OxtGxHLgUuChwEmZeVFEfBDYn6bAvzczP97p97v1jnXcdNta9pzYsSvjlSRJkuajm0ekNwLHAP8047EzM/Mw4Ejg5IjouMjvutMK9thl5QIPUZIkSarTtSKdmVOZ+ZtZj/1n++F6YBMw1en3O+3Fh7J82aKZiSJJkqQh17WpHQWvBi7MzE2dPmFi9+GZ0jE2tpTxcY+uz8WMysyozIzKzKjMjMrMqMyMyhZjRj0v0hFxDPBo4Lnzed7k5NruDKgPxsdXDtXP0w1mVGZGZWZUZkZlZlRmRmVmVDYIGU1MrJrX8j0t0hHxOODlwNMy00twSJIkadHqapGOiAuBg4E7I+JQ4C+BNcBXIgLg2My8uZtjkCRJkrqhq0U6M2dP33hdN9cnSZIk9YqXwZAkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSapgkZYkSZIqWKQlSZKkChZpSZIkqYJFWpIkSaqwrFvfOCKWA5cCDwVOysyLImJ34OPAKuBrmfnmbq1fkiRJ6qZuHpHeCBwD/NOMx14PnJuZRwCHRMR+XVy/JEmS1DVdK9KZOZWZv5n18BHAF9uPvwgc2a31S5IkSd3UtakdW7FDZv6h/XgS2KvTJ46Pr+zOiPpgbGzpUP083WBGZWZUZkZlZlRmRmVmVGZGZYsxo14X6bURsSIz1wE7A7d2+sTJybXdG1WPjY+vHKqfpxvMqMyMysyozIzKzKjMjMrMqGwQMpqYWDWv5Xt91Y7Lgae2Hz+l/VySJEladLp6RDoiLgQOBu6MiEOBM4GPRcTfA1/PzB92c/2SJElSt3S1SGfmc7fw8FO6uU5JkiSpF7whiyRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUgWLtCRJklTBIi1JkiRVsEhLkiRJFSzSkiRJUoVlvV5hRLwXOJCmxP9DZl7a6zFIkiRJ26qnR6QjYh9g38w8HDgWOL2X65ckSZIWSq+ndtwE/CEilgHjwM09Xr8kSZK0IHo9tWMNcAOQwErg2Z0+cXx8ZbfG1HNjY0uH6ufpBjMqM6MyMyozozIzKjOjMjMqW4wZ9bpIPxHYDdgH2AP4InBQJ0+cnFzbxWH11vj4yqH6ebrBjMrMqMyMysyozIzKzKjMjMoGIaOJiVXzWr7XUzuWArdm5mbgDmCHHq9fkiRJWhC9LtL/BqyKiMuBS4G39Hj9kiRJ0oLo6dSOzNwEnNDLdUqSJEnd4A1ZJEmSpAoWaUmSJKmCRVqSJEmq0FGRjogDI+K49uNdI2LP7g5LkiRJGmzFkw0j4o3AwcC+wAXAvYDzgSO6OzRJkiRpcHVyRPo5wLOAuwAy81fAjt0clCRJkjToOinSG9q/pwAiYufuDUeSJElaHDop0mfTTOXYLSL+AbgMeGdXRyVJkiQNuOIc6cz8SER8CzgaWAIcn5k/7PrIJEmSpAFWPCIdEQ8Bfp6Z7wV+CBwdEbt2fWSSJEnSAOtkaseFwMaI2Bd4DzBOM9VDkiRJGlmdFOnNmbkJOAZ4T2aeDuze3WFJkiRJg604RxpYGxGnAicAR0bEUmC77g5LkiRJGmydHJE+FlgDnJiZvwb2BM7s6qgkSZKkAVcs0pn5G+Ac4NftiYcrgG91e2CSJEnSIOvkFuFvBk4Erqe9KUv79+O7NyxJkiRpsHUyR/qvgH0yc323ByNJkiQtFp3Mkf4OcN9uD0SSJElaTDo5Iv1+4JqIuB5YT3N3w6nMPLyrI5MkSZIGWCdF+sM0c6R/AGzu7nAkSZKkxaGTIv27zPx810ciSZIkLSKdFOmfRMSXgK/QTO0AIDM/0LVRSZIkSQOukyL9y/bPbjMem9rKspIkSdJImLNIt7cD3zEzX9uj8UiSJEmLwpyXv8vMzcDhbaGWJEmS1OpkakcCX4+Ii4G19zzoHGlJkiSNsE6K9A3tn53bP5IkSdLIKxbpzDytFwORJEmSFpOO5j5HxEvm+lySJEkaNZ2eRLik8LkkSZI0UopFOiL2ycz3z3r4610ajyRJkrQodHJE+oItPHb+Qg9EkiRJWky2erJhRBwIHALsNmtO9E5zPU+SJEkaBXMV4lXAfYDtgPvOeHwNcEztCiPiUOAMYDnw5cw8q/Z7SZIkSf2y1SKdmZcBl0XEhzPzRrjnluGrMvP2mpVFxPbAqcCzMnNtaXlJkiRpUC2Zmpqac4GIuAA4qf302zTl+6OZ+bb5riwijgJeCaxsv89rMvP7HTx1asOGTfNd3cAaG1vKpk2b+z2MgWZGZWZUZkZlZlRmRmVmVGZGZYOQ0fLlYzCPq9N1Mtf5IZm5JiJOBD4HnAxcC8y7SAP3A/ajmXt9f+CDwBGdPHFycngOYI+Prxyqn6cbzKjMjMrMqMyMysyozIzKzKhsEDKamFg1r+U7uWrHdhHxAOD5wOcyc1t+VbgNuCIz78rMH+MtxyVJkrRIdXJE+i3AF4BvZua3ImIv4KeV6/s28IZ2rvW9gXWV30eSJEnqq2KRzszzI+Ii2it3ZOb1wHNqVpaZt0XER4HLaK7a8eqa7yNJkiT1W7FIR8TTgHcAOwB7RcQjgDdmZtUl8DLzXODcmudKkiRJg6KTOdJvAQ4DJgEy83tAdHNQkiRJ0qDrpEhvzMw1059ExBJg7mvmSZIkSUOukyK9OiL+lubqHY8G/gW4pLvDkiRJkgZbJ0X6RuBu4MfAa4H/AG7u5qAkSZKkQdfJ5e+en5kHAu+ffiAirgXO6tqoJEmSpAG31SIdEf+T5tbgD4mI1TO+tAr4QbcHJkmSJA2yuY5IXwT8O3AGzW3Bp63JzFu7OipJkiRpwG21SGfm7cDtwAt6NxxJkiRpcejkZENJkiRJs1ikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqsKwfK42II4DLgYnMvKUfY5AkSZK2Rb+OSL8KuLpP65YkSZK2Wc+LdEQ8HfgmcFev1y1JkiQtlCVTU1M9W1lELAW+CDwHuAQ4psOpHVMbNmzq6th6aWxsKZs2be73MAaaGZWZUZkZlZlRmRmVmVGZGZUNQkbLl48BLOl0+V7PkX4+8PnMXBcR83ri5OTa7oyoD8bHVw7Vz9MNZlRmRmVmVGZGZWZUZkZlZlQ2CBlNTKya1/K9ntrxMOCYiLgEeDjwyR6vX5IkSVoQPT0inZmvn/44Ii6lOUItSZIkLTp9ufwdQGY+rl/rliRJkraVN2SRJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqsKyXK4uIw4D/DawH7gSOz8zJXo5BkiRJWgi9PiJ9A3B0Zh4FfAH4Xz1evyRJkrQgenpEOjN/PePT9cDGXq5fkiRJWihLpqamer7SiNgN+Crw5My8pYOnTG3YsKnLo+qdsbGlbNq0ud/DGGhmVGZGZWZUZkZlZlRmRmVmVDYIGS1fPgawpNPle3pEGiAiVgKfBv62wxINwOTk2u4NqsfGx1cO1c/TDWZUZkZlZlRmRmVmVGZGZWZUNggZTUysmtfyPZ0jHRHLgPOB92Tm6l6uW5IkSVpIvT7Z8HnAkcDfRcSlEfHaHq9fkiRJWhC9Ptnw48DHe7lOSZIkqRu8IYskSZJUwSItSZIkVbBIS5IkSRUs0pIkSVIFi7QkSZJUwSItSZIkVbBIS5IkaSht2LiZG2++kw0bu3Pr8Z7fIrzWjTffyR67rGT5Mru/JEmS5rZh42ZOPfdKbr1jHbvutILTXnzogvfIRdNKz/iXqzn13Cu79huFJEmShsdNt63l1jvWsX7jZm69Yx033bZ2wdexaIp0N0OQJEnScNljl5XsutMKtlu2lF13WsEeu6xc8HUsmqkd3QxBklS2YeNmbrptrdPsJC0Ky5ct5bQXH9rV161FU6RP+euDffGWpD7pZK6hRVvbwu1H3bB82VL2nNixo2Vrpg8vmi11z4kd/Y+lRaXTM4W7fUaxtBBKcw2ni7bns6hTM1/73H7Ub9Pb4HwtmiPS0mLS6ZnCvTijWFoI03MNp7fV2dPstlS0Oz0KpNEz+7XvJc/Yz+1HfTX9GjZfFmmpCzotFZYPLRaluYaloi3NNPu1jyW4/aivpl/D5ssiLXVBp6XC8qHFZK65hr04qUfDY/Zr35/tvqPbj/pq+jVsvpZMTU11YTgLburmm9f0ewwLZnx8JZOTXsZvLsOQUacnztSeYDMMGXWbGZWZUZkZldVkNGonF7odlQ1CRhMTqwCWdLq8R6SlLun0TOH5nFEsScPC1z4Ng+H/FVCSJEnqAou0JEmSVMEiLUmSJFWwSEuSJEkVLNKSJElSBYu0JEmSVMEiLUmSJFVYNDdk6fcAJEmSNBKG7oYsHf9AkiRJUi84tUOSJEmqYJGWJEmSKlikJUmSpAoWaUmSJKmCRVqSJEmqYJGWJEmSKlikJUmSpAoW6S6LCK+BXWBGWghuR1oIbkdlZlRmRqNjsdyQZVGJiAcCzwC+CPwS2NTXAQ2giNgLOBU4JTNv7Pd4BlG7HT0KuCwzf9vn4Qykdjt6BXBmZt7U7/EMIrejMrejMvdrZe7Xyobx9cgj0gssIo4CPgXsCbwGOL6/Ixo8EfEq4DPAl3yx2bKIeCrwWeDRwLsi4vA+D2mgRMSSiHgtcAGw2vKzZW5Hc3M76oz7tTL3a2XD+npkkV542wMXZ+brgY8DD4uIZ/Z5TAMjInYFdgM+BqyJiFdHxFERsbzPQxs0q2iOjr0K+ApwdEQc1ucxDZL7A/sCbwPuiIgXRsRBfR7TINoJt6O5PADYD7ejEvdrc4iIXXC/1omh3K9ZpLdRRDw2Is6JiCdGxCpgB/44ZeZa4BvAURGxom+D7LOZGQF30/xG+j+AvwMmad5SPT4ituvjMPsqIg6JiENmPLQfsEf78deA3wIHR8TITsdqMzoUIDNvAC4HXkKzHY0DZ7U7r5F9XYuIR0XEmRGxf5vDA4D7tl92O+KejM6KiP0z8xc0O/SX4XZ0j4h4dER8OCKe3e7XVvHHvuB+jXsyOjcing1sAC4GHof7tXtExO4RcWVE3K99aCj3ayP7QrEQImJfmre5VgNHAW8H/hU4IiL2ysx1wM9oyuOufRtoH83K6Ejg3Zn5HeDNmfmUzPww8AHgL2herEdORJwGfBt4bkT8RfvwBcAJEbGynUf2G+DewKZRPIllRkbHRkS0D38WeHtmPj0z/y/wSeBJmbm5X+Psl4hYHhFn0bwG3QG8AHgWzXZ0fETsMOrb0ayMbqfJ5Xk0Beitbkf3THV5B/CPwNXAI4HjgCuAx7pf+28ZXQUcALwwM68GTnW/9id2Ah4MPK/9fHq/NlSvRxbpCjP+wfcANmfmxzLzFOAQYG/g08DLI+I+wA3Ag2heuEfGVjJ6I81bgo/PzGtmvO11HU1Gd/ZjrP0y44jXp4ADgd/R5LMyM39Ec9Tn9HaZbwMPAbbLzKmeD7ZPtpLRwyNiRWZOAlfM2Na+Duw1om+nrqCZ4/v4zDwD+BGwQ3vE9XLgtHa5kdyOWrMz+jGwNDPXA6tHfTuKiCXtNnFRZj4uM88GrgRub0vPV3G/tqWMrgJuAsjMa0d9vzbLRuBS4BkRcUi7X/smQ/Z6ZJGep4hYOuMf/PvAjdNvNwNnAv+YmefSnNX8JuBLwE+BDYv5N675KGT0DuCU9uONEXEscCHwPRb5b6Xz0Wa0GSAzf5KZ1wI/p3nra/92sVOA7SPincDngO8wQmfKd5JR+/UlEfFcmvmJqxmtjMYAMnMNTdGZtivNkSCANwDbjfB2NFdGe7df28SIb0fTr9mZeVX72KOAjwLPjoizM/P/MNr7tbkyelZEfLT92oZ2OxrJ/dqsh3YEzgc+Ary8fex1wLKIeBdD8nq0ZGpq0f4S0DMRcQBwAnByZq6LiPsDD6eZW/dyYAlwdvsf6CKayfRXRsTeNEeFvt+3wffIPDO6EHg38HuauYkXZebqPg29Z7aQ0Z8DB2bm59qv7wC8muZt009Nl8iIeASwvH3rcKjNJ6PM/ET72P40L85nZ+Z/9GnoPTNXRtNHzNqd+jsz87oZz3M72kpG7VHE6WlobkdN8XsksCIzV0fEJTT7ta+P+H5troy+TLNfu55mnzfK+7WDMvPiiNgHeH1mnhQR/0ZzDsJLaebZ70eT3aJ/PVrUE7y7LSK2B94CHEozh25d+6V1wKbM3BQR3wWeBvx1RJwL3EbzmzqZ+bM+DLunKjO6HfhJZt4G/H0/xt1LhYzWt8ssycy7IuLTNCfQPTMirmuX/14/xt1LNRlFc9WA72fmW4EX9mPcvdRJRjSv6RuAXwNjEXE6zTzEc9yO5szoV5n5ftyO1gO0R16vaZffieaX+5+1Xxv1/drWMrqeZr/2X7hfu7v9eC3wX+27YTsA98nMa9qvXceQsEjPLWjOeD8d2CEiXgBclZlJc6QVmrcAb6LZoJ5L8zbFbX0Ya7/UZjQ5Y77ZsOskoyXAFM2JK8+kucTU20fohKfajN7Rh7H2SzGj9h2fnYFXAkcDnwc+MCL/z6A+ow/1abz9UMyoPdo6BvwNzWv2N2imdYyK2oxudL/2J6/Z0JwnlZl5eEQ8Hf5krvlQcGrHLO0/9OHAeZn5o4h4Gc0ZpzfSnMBzCPC2zLyyXX7XzLw1InanOQI79CXajMoqMtolM2+LiMcCv8zm8m5DzYzKKjIapzlA8krgI9mcbDjUzKisIqOdaI4mngh8LTOv78/Ie8eMyioy2j4z797qNxwSFukZIuI5wItofpvaF/jPzHxPRDwmM69ol3kdTRmcvivPkzLz1L4NusfMqKwyoydn5pv6NugeM6OyioweAzw+M9/St0H3mBmVVf5fe2JmnrbVbzpkzKhsW/b9w3YEejav2sGfXKptguatibNp3q54cUTsk5lXxB8vGP4dmkvakJmrR6UgmlHZNmY0EgXRjMq2IaMrRqUgmlHZNv5fG4mCaEZlC7HvH+YSDSNepKO5AsBMPwXWR8R9M/MWmku2TO+8xyLiJJqL+X+jff7QX9LGjMrMqMyMysyozIzKzKjMjMrMqHMjWaSjuf3rV4CD24em/8HvAO5Fc1kWgA/SXOt4d5qTUg4FTszMC2C4f8syozIzKjOjMjMqM6MyMyozozIzmr+RumpHREzQ3FFnguZs08OAy6a/nplXRcQjaW6FehvNiQR3tb99fbn9M9TMqMyMysyozIzKzKjMjMrMqMyM6o3aEemDgO9m5rHAscCDI+Jembl5xtsQFwBXAycD59Be63CE3qYwozIzKjOjMjMqM6MyMyozozIzqjT0V+2I5lad+wCfzBmXp4mIJwBPoNkgpnLW9XojYi/glmxuKzvUzKjMjMrMqMyMysyozIzKzKjMjBbG0BbpiLg38GHgTprbUe4BfDYzL2+/vgfw/4Aj27cmRo4ZlZlRmRmVmVGZGZWZUZkZlZnRwhr2qR0fysznAe8Efk9za2oiYiwzbwK+RPNb1ygzozIzKjOjMjMqM6MyMyozozIzWiDDXKRvBi4ByMxNwCOAe09/HhHLgZU0v5GNKjMqM6MyMyozozIzKjOjMjMqM6MFNLRX7WgvvXJ3RIwB2wE7ZubXANoJ9H+IiC8A3+rnOPvJjMrMqMyMysyozIzKzKjMjMrMaGENbZGeYTOwPXB1RBwMnEBzi8tLMvOSvo5scJhRmRmVmVGZGZWZUZkZlZlRmRktgKE92XCmiHgSzTUO/xX4RGae1+chDRwzKjOjMjMqM6MyMyozozIzKjOjbTcKR6QBJoE3AO/OzPX9HsyAMqMyMyozozIzKjOjMjMqM6MyM9pGo1Kkr8zMb/d7EAPOjMrMqMyMysyozIzKzKjMjMrMaBuNxNQOSZIkaaEN8+XvJEmSpK6xSEuSJEkVLNKSJElSBYu0JEmSVGFUrtohSdssIq6med28D7AB+D1wS2Y+oYPnPgp4ZmaePMcyzwbul5nvqxjbKuBcINox/iwznxERDwIOyMzPzPP7PR6YzMxrIuJ04Jk0d0G7P/DTdrGXAqcDT21vNSxJI8WrdkjSPEXEm4HfZuY5sx4f61ehjIg3AWTm6e3nD8vMH0TEE4AXZeYL5vG9xoDTgF9k5odmPP5g4KOZecTCjl6SFiePSEvSNmiL6uuAKWAsIo4FPgPs3C7yisxcPbPQRsQZwASwP7AH8NLM/PeIOAl4YGaeEhHnAbcAhwM7Ase1xfg+wIXALjS3831OZj6Y5ij5d6bHlZk/aD98K7BPRFwLvB24EXgXza2Bfw8cn5k3tWO6H7AfcANwFLA+Il4B/GVm/mwrP/+NwAPbP+cDPwcOAs4DfgW8DFgHPDkz74iIAN7Xjv93wAsz8+b5pS5Jg8E50pK07Q4ETmineNwFPCMzDwSeRVNat+TPgSOB44E3bWWZe2XmoTRHh1/TPnY68MnMfBjw6xnLfgR4R0RcFhFvjIg/ax8/GfhyZh6QmRcA1wGPycxH0kwFec2M7/FA4IjMPA74EHB6+7wtlugt2A94LfBQ4G+AHdocrgaObZd5H3BiZh4EfKwdnyQtShZpSdp238jM37UfLwHeFRE/AD5PUy635EvtNJDv0hTYLfl8+/fMZQ4HLmg/nv6bzLwK2Bv4Z2Af4JqI2G0L33MX4OJ2fG+aNb6LM3PjVsbSiesy84bMXAf8Avjq9OPA/SNiHDgM+EJ7hPwNwJ7bsD5J6iundkjStls74+MXtn8fAGwG7tjKc+5u/94MjM1jmSVbG0Rm3k5Tri+IiEuAxwJ3zlrsrcAFmXleRDwaOGMrP0eN9TM+3jzj8+nxLwVuyMwDtnE9kjQQPCItSQtrJ+B37dHm44AVC/z9V/PHaRLTfxMRR0TEzu3Hq2iOYP8SWAOsmjW+37Qfv2iO9cx+3jbLzFuBP0TE0e04t4uIfRdyHZLUSxZpSVpY5wFPjojvA4+iOaFvIZ0KnNBOzXgQfzzivQ/wzXa93wI+mJnX0EwL2Tkiro2I44CzgHMi4hrg9jnW8zngxe3z9l7A8T8PeENEfA+4Bjh4Ab+3JPWUl7+TpEUkIlYAGzJzU0S8iObkwJP6PCxJGknOkZakxWVv4BMRsZTmaPcJfR6PJI0sj0hLkiRJFZwjLUmSJFWwSEuSJEkVLNKSJElSBYu0JEmSVMEiLUmSJFX4/6qgbEOSupdbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# analyze using seaborn\n",
    "plt = df_fm_tuner_metrics.plot(kind='line', figsize=(12,5), x='TrainingStartTime', \n",
    "                             y='FinalObjectiveValue', \n",
    "                             style='b.', legend=False)\n",
    "plt.set_ylabel(objective_metric_name);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Best Factorization Machine Model after Hyper-Parameter Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fm_tuner_job_name: hpo-fm-20201027-045403\n",
      "fm_best_model_name: hpo-fm-20201027-045403-003-ba3bbbca\n"
     ]
    }
   ],
   "source": [
    "print(\"fm_tuner_job_name: \" + fm_tuner_job_name)\n",
    "fm_tuner = HyperparameterTuner.attach(fm_tuner_job_name)\n",
    "\n",
    "fm_tuner_analytics = HyperparameterTuningJobAnalytics(hyperparameter_tuning_job_name=fm_tuner_job_name)\n",
    "df_fm_tuner_metrics = fm_tuner_analytics.dataframe()\n",
    "\n",
    "fm_best_model_name = fm_tuner.best_training_job()\n",
    "print(\"fm_best_model_name: \" + fm_best_model_name)\n",
    "\n",
    "fm_model_info = sm.describe_training_job(TrainingJobName=fm_best_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>factors_init_sigma</th>\n",
       "      <th>factors_lr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.061027</td>\n",
       "      <td>511.0</td>\n",
       "      <td>2020-10-27 05:16:59+00:00</td>\n",
       "      <td>hpo-fm-20201027-045403-003-ba3bbbca</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-10-27 05:08:28+00:00</td>\n",
       "      <td>0.016572</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FinalObjectiveValue  TrainingElapsedTimeSeconds           TrainingEndTime  \\\n",
       "7             1.061027                       511.0 2020-10-27 05:16:59+00:00   \n",
       "\n",
       "                       TrainingJobName TrainingJobStatus  \\\n",
       "7  hpo-fm-20201027-045403-003-ba3bbbca         Completed   \n",
       "\n",
       "          TrainingStartTime  factors_init_sigma  factors_lr  \n",
       "7 2020-10-27 05:08:28+00:00            0.016572      0.0001  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_fm_tuner_metrics[df_fm_tuner_metrics['TrainingJobName']==fm_best_model_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's evaluate the results with the best training job from hyper-parameter tuning job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can deploy the endpoint using hyper-parameter tuning job and test the predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-10-27 05:16:59 Starting - Preparing the instances for training\n",
      "2020-10-27 05:16:59 Downloading - Downloading input data\n",
      "2020-10-27 05:16:59 Training - Training image download completed. Training in progress.\n",
      "2020-10-27 05:16:59 Uploading - Uploading generated training model\n",
      "2020-10-27 05:16:59 Completed - Training job completed\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34mRunning default environment configuration script\u001b[0m\n",
      "\u001b[34m/opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.\n",
      "  from numpy.testing import nosetester\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'factors_lr': u'0.0001', u'predictor_type': u'regressor', u'factors_init_sigma': u'0.01657242760253564', u'_tuning_objective_metric': u'test:rmse', u'epochs': u'20', u'feature_dim': u'178729', u'bias_lr': u'0.02', u'num_factors': u'512', u'mini_batch_size': u'200'}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'20', u'feature_dim': u'178729', u'num_factors': u'512', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.01657242760253564', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'test:rmse', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.02', u'mini_batch_size': u'200', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'regressor', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 WARNING 139876871620416] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34mProcess 1 is a worker.\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Using default worker.\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:09:19.529] [tensorio] [warning] TensorIO is already initialized; ignoring the initialization routine.\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:09:19.531] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 0, \"duration\": 4, \"num_examples\": 1, \"num_bytes\": 13400}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] nvidia-smi took: 0.0251140594482 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:19 INFO 139876871620416] Create Store: local\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 33.91885757446289, \"sum\": 33.91885757446289, \"min\": 33.91885757446289}}, \"EndTime\": 1603775359.564579, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775359.526763}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 200, \"sum\": 200.0, \"min\": 200}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 200, \"sum\": 200.0, \"min\": 200}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603775359.564707, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775359.564683}\n",
      "\u001b[0m\n",
      "\u001b[34m[05:09:19] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.203343.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[05:09:19] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.203343.0/AL2012/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=0 train rmse <loss>=4.24104554151\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=0 train mse <loss>=17.9864672852\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=0 train absolute_loss <loss>=4.08906066895\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:23 INFO 139876871620416] Iter[0] Batch [500]#011Speed: 41762.06 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=500 train rmse <loss>=2.03172383376\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=500 train mse <loss>=4.12790173666\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=500 train absolute_loss <loss>=1.65503928407\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:25 INFO 139876871620416] Iter[0] Batch [1000]#011Speed: 44342.73 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1000 train rmse <loss>=1.66733819602\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1000 train mse <loss>=2.7800166599\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1000 train absolute_loss <loss>=1.30170746833\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:28 INFO 139876871620416] Iter[0] Batch [1500]#011Speed: 45768.48 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1500 train rmse <loss>=1.52343666694\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1500 train mse <loss>=2.32085927817\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=1500 train absolute_loss <loss>=1.18049287947\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:30 INFO 139876871620416] Iter[0] Batch [2000]#011Speed: 45327.54 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2000 train rmse <loss>=1.44420855388\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2000 train mse <loss>=2.0857383471\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2000 train absolute_loss <loss>=1.11816012831\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:32 INFO 139876871620416] Iter[0] Batch [2500]#011Speed: 44421.53 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2500 train rmse <loss>=1.39347338351\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2500 train mse <loss>=1.94176807054\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=2500 train absolute_loss <loss>=1.07966984877\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:34 INFO 139876871620416] Iter[0] Batch [3000]#011Speed: 43827.12 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3000 train rmse <loss>=1.35812884782\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3000 train mse <loss>=1.84451396727\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3000 train absolute_loss <loss>=1.05346770861\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] Iter[0] Batch [3500]#011Speed: 45665.21 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3500 train rmse <loss>=1.33088919842\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3500 train mse <loss>=1.77126605847\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, batch=3500 train absolute_loss <loss>=1.0335445651\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:09:37.123] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 2, \"duration\": 16922, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, train rmse <loss>=1.33108056937\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, train mse <loss>=1.77177548215\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=0, train absolute_loss <loss>=1.03361844583\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 20, \"sum\": 20.0, \"min\": 20}, \"update.time\": {\"count\": 1, \"max\": 17559.245824813843, \"sum\": 17559.245824813843, \"min\": 17559.245824813843}}, \"EndTime\": 1603775377.124081, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775359.564639}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #progress_metric: host=algo-1, completed 5 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 3508, \"sum\": 3508.0, \"min\": 3508}, \"Total Records Seen\": {\"count\": 1, \"max\": 701541, \"sum\": 701541.0, \"min\": 701541}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1603775377.124235, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1603775359.564813}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=39940.8504057 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=0 train rmse <loss>=1.09071065572\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=0 train mse <loss>=1.1896497345\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=0 train absolute_loss <loss>=0.866179351807\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:39 INFO 139876871620416] Iter[1] Batch [500]#011Speed: 44893.70 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=500 train rmse <loss>=1.1432588473\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=500 train mse <loss>=1.30704079192\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=500 train absolute_loss <loss>=0.907871933593\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:41 INFO 139876871620416] Iter[1] Batch [1000]#011Speed: 46371.35 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:41 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1000 train rmse <loss>=1.13861089269\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:41 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1000 train mse <loss>=1.29643476496\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:41 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1000 train absolute_loss <loss>=0.902332116888\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:43 INFO 139876871620416] Iter[1] Batch [1500]#011Speed: 44372.27 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:43 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1500 train rmse <loss>=1.13460304023\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:43 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1500 train mse <loss>=1.28732405889\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:43 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=1500 train absolute_loss <loss>=0.898194343674\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:45 INFO 139876871620416] Iter[1] Batch [2000]#011Speed: 44903.82 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:45 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2000 train rmse <loss>=1.13122883402\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:45 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2000 train mse <loss>=1.27967867491\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:45 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2000 train absolute_loss <loss>=0.895518146765\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:48 INFO 139876871620416] Iter[1] Batch [2500]#011Speed: 45699.03 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2500 train rmse <loss>=1.12873443936\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2500 train mse <loss>=1.2740414346\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=2500 train absolute_loss <loss>=0.893538146906\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:50 INFO 139876871620416] Iter[1] Batch [3000]#011Speed: 47282.80 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:50 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3000 train rmse <loss>=1.12715653937\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:50 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3000 train mse <loss>=1.27048186425\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:50 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3000 train absolute_loss <loss>=0.892186406691\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] Iter[1] Batch [3500]#011Speed: 46268.00 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3500 train rmse <loss>=1.12488175887\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3500 train mse <loss>=1.26535897145\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, batch=3500 train absolute_loss <loss>=0.8903734975\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:09:52.487] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 4, \"duration\": 15362, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, train rmse <loss>=1.12540761059\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, train mse <loss>=1.26654228998\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=1, train absolute_loss <loss>=0.890626992366\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15363.94715309143, \"sum\": 15363.94715309143, \"min\": 15363.94715309143}}, \"EndTime\": 1603775392.488364, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775377.124144}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #progress_metric: host=algo-1, completed 10 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 7015, \"sum\": 7015.0, \"min\": 7015}, \"Total Records Seen\": {\"count\": 1, \"max\": 1402882, \"sum\": 1402882.0, \"min\": 1402882}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1603775392.488514, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1603775377.124394}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=45647.7370831 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=0 train rmse <loss>=1.06329265871\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=0 train mse <loss>=1.13059127808\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:52 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=0 train absolute_loss <loss>=0.846711273193\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:54 INFO 139876871620416] Iter[2] Batch [500]#011Speed: 45225.95 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=500 train rmse <loss>=1.10177547934\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=500 train mse <loss>=1.21390920689\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=500 train absolute_loss <loss>=0.875303196555\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:56 INFO 139876871620416] Iter[2] Batch [1000]#011Speed: 43963.46 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1000 train rmse <loss>=1.09762902464\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1000 train mse <loss>=1.20478947574\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1000 train absolute_loss <loss>=0.869953089665\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:59 INFO 139876871620416] Iter[2] Batch [1500]#011Speed: 46396.03 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:59 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1500 train rmse <loss>=1.09452489035\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:59 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1500 train mse <loss>=1.19798473559\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:09:59 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=1500 train absolute_loss <loss>=0.866197142985\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:01 INFO 139876871620416] Iter[2] Batch [2000]#011Speed: 45545.76 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:01 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2000 train rmse <loss>=1.09193705809\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:01 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2000 train mse <loss>=1.19232653883\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:01 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2000 train absolute_loss <loss>=0.863992608226\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:03 INFO 139876871620416] Iter[2] Batch [2500]#011Speed: 43204.16 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2500 train rmse <loss>=1.09020567946\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2500 train mse <loss>=1.18854842353\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=2500 train absolute_loss <loss>=0.862525261624\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:05 INFO 139876871620416] Iter[2] Batch [3000]#011Speed: 45317.89 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3000 train rmse <loss>=1.08938257057\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3000 train mse <loss>=1.18675438507\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3000 train absolute_loss <loss>=0.861659738273\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] Iter[2] Batch [3500]#011Speed: 44690.56 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3500 train rmse <loss>=1.08783322798\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3500 train mse <loss>=1.18338113189\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, batch=3500 train absolute_loss <loss>=0.860308498915\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:10:08.117] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 6, \"duration\": 15627, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, train rmse <loss>=1.08837671952\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, train mse <loss>=1.1845638836\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=2, train absolute_loss <loss>=0.860565566625\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15629.048824310303, \"sum\": 15629.048824310303, \"min\": 15629.048824310303}}, \"EndTime\": 1603775408.118057, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775392.488426}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #progress_metric: host=algo-1, completed 15 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 10522, \"sum\": 10522.0, \"min\": 10522}, \"Total Records Seen\": {\"count\": 1, \"max\": 2104223, \"sum\": 2104223.0, \"min\": 2104223}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1603775408.118204, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1603775392.488984}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44873.4546311 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=0 train rmse <loss>=1.04166479085\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=0 train mse <loss>=1.0850655365\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:08 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=0 train absolute_loss <loss>=0.830214004517\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:10 INFO 139876871620416] Iter[3] Batch [500]#011Speed: 42543.12 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=500 train rmse <loss>=1.07018294713\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=500 train mse <loss>=1.14529154033\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=500 train absolute_loss <loss>=0.849304413025\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:12 INFO 139876871620416] Iter[3] Batch [1000]#011Speed: 43469.87 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1000 train rmse <loss>=1.06611216566\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1000 train mse <loss>=1.13659514977\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1000 train absolute_loss <loss>=0.843767311409\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:15 INFO 139876871620416] Iter[3] Batch [1500]#011Speed: 44794.46 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1500 train rmse <loss>=1.06337446804\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1500 train mse <loss>=1.13076525928\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=1500 train absolute_loss <loss>=0.840148566625\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:17 INFO 139876871620416] Iter[3] Batch [2000]#011Speed: 44136.18 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2000 train rmse <loss>=1.06110181579\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2000 train mse <loss>=1.12593706347\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2000 train absolute_loss <loss>=0.83808610755\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:19 INFO 139876871620416] Iter[3] Batch [2500]#011Speed: 43293.94 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2500 train rmse <loss>=1.05969316636\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2500 train mse <loss>=1.12294960684\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=2500 train absolute_loss <loss>=0.83682592289\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:21 INFO 139876871620416] Iter[3] Batch [3000]#011Speed: 42811.23 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3000 train rmse <loss>=1.05920248877\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3000 train mse <loss>=1.12190991222\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3000 train absolute_loss <loss>=0.83615706467\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] Iter[3] Batch [3500]#011Speed: 43573.74 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3500 train rmse <loss>=1.05798625197\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3500 train mse <loss>=1.11933490935\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, batch=3500 train absolute_loss <loss>=0.835006716286\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:10:24.242] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 8, \"duration\": 16123, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, train rmse <loss>=1.05854351535\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, train mse <loss>=1.12051437389\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=3, train absolute_loss <loss>=0.835264514263\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16124.869108200073, \"sum\": 16124.869108200073, \"min\": 16124.869108200073}}, \"EndTime\": 1603775424.243254, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775408.118115}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #progress_metric: host=algo-1, completed 20 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14029, \"sum\": 14029.0, \"min\": 14029}, \"Total Records Seen\": {\"count\": 1, \"max\": 2805564, \"sum\": 2805564.0, \"min\": 2805564}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 5, \"sum\": 5.0, \"min\": 5}}, \"EndTime\": 1603775424.243403, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 3}, \"StartTime\": 1603775408.118361}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43493.6778857 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=0 train rmse <loss>=1.0216581622\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=0 train mse <loss>=1.04378540039\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=0 train absolute_loss <loss>=0.813530273437\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:26 INFO 139876871620416] Iter[4] Batch [500]#011Speed: 43578.19 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=500 train rmse <loss>=1.04302739277\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=500 train mse <loss>=1.08790614206\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=500 train absolute_loss <loss>=0.826067116256\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:28 INFO 139876871620416] Iter[4] Batch [1000]#011Speed: 45722.46 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1000 train rmse <loss>=1.03886573659\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1000 train mse <loss>=1.07924201866\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1000 train absolute_loss <loss>=0.820208522557\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:30 INFO 139876871620416] Iter[4] Batch [1500]#011Speed: 45357.21 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1500 train rmse <loss>=1.03629140397\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1500 train mse <loss>=1.07389987394\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=1500 train absolute_loss <loss>=0.816606531264\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:33 INFO 139876871620416] Iter[4] Batch [2000]#011Speed: 45071.03 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2000 train rmse <loss>=1.0341443972\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2000 train mse <loss>=1.06945463426\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2000 train absolute_loss <loss>=0.814548700801\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:35 INFO 139876871620416] Iter[4] Batch [2500]#011Speed: 44269.37 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2500 train rmse <loss>=1.03287858649\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2500 train mse <loss>=1.06683817443\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=2500 train absolute_loss <loss>=0.813368107194\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:37 INFO 139876871620416] Iter[4] Batch [3000]#011Speed: 43624.67 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3000 train rmse <loss>=1.03254273686\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3000 train mse <loss>=1.06614450345\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3000 train absolute_loss <loss>=0.812771243051\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:39 INFO 139876871620416] Iter[4] Batch [3500]#011Speed: 43865.29 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3500 train rmse <loss>=1.03148856583\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3500 train mse <loss>=1.06396866143\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, batch=3500 train absolute_loss <loss>=0.811707769752\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:10:40.014] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 10, \"duration\": 15769, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, train rmse <loss>=1.03205809004\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, train mse <loss>=1.06514390121\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=4, train absolute_loss <loss>=0.811965672015\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15771.010160446167, \"sum\": 15771.010160446167, \"min\": 15771.010160446167}}, \"EndTime\": 1603775440.014592, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775424.243313}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #progress_metric: host=algo-1, completed 25 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 17536, \"sum\": 17536.0, \"min\": 17536}, \"Total Records Seen\": {\"count\": 1, \"max\": 3506905, \"sum\": 3506905.0, \"min\": 3506905}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 6, \"sum\": 6.0, \"min\": 6}}, \"EndTime\": 1603775440.014736, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 4}, \"StartTime\": 1603775424.243558}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44469.5664388 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=0 train rmse <loss>=1.00173913407\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=0 train mse <loss>=1.00348129272\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=0 train absolute_loss <loss>=0.796387710571\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:42 INFO 139876871620416] Iter[5] Batch [500]#011Speed: 45051.92 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=500 train rmse <loss>=1.01786610062\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=500 train mse <loss>=1.03605139879\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=500 train absolute_loss <loss>=0.803856589494\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:44 INFO 139876871620416] Iter[5] Batch [1000]#011Speed: 45995.85 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1000 train rmse <loss>=1.01353356832\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1000 train mse <loss>=1.02725029411\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1000 train absolute_loss <loss>=0.797604599837\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:46 INFO 139876871620416] Iter[5] Batch [1500]#011Speed: 46008.34 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1500 train rmse <loss>=1.01102888745\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1500 train mse <loss>=1.02217941127\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=1500 train absolute_loss <loss>=0.793956785869\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:48 INFO 139876871620416] Iter[5] Batch [2000]#011Speed: 44759.87 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2000 train rmse <loss>=1.00891638362\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2000 train mse <loss>=1.01791226915\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:48 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2000 train absolute_loss <loss>=0.791834244087\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:51 INFO 139876871620416] Iter[5] Batch [2500]#011Speed: 44601.62 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2500 train rmse <loss>=1.00770390029\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2500 train mse <loss>=1.01546715066\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=2500 train absolute_loss <loss>=0.790663963708\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:53 INFO 139876871620416] Iter[5] Batch [3000]#011Speed: 42336.14 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3000 train rmse <loss>=1.00743401556\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3000 train mse <loss>=1.0149232957\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3000 train absolute_loss <loss>=0.790071606774\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] Iter[5] Batch [3500]#011Speed: 45232.96 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3500 train rmse <loss>=1.00645465476\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3500 train mse <loss>=1.01295097209\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, batch=3500 train absolute_loss <loss>=0.789029666091\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:10:55.665] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 12, \"duration\": 15649, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, train rmse <loss>=1.00703629859\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, train mse <loss>=1.01412210667\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=5, train absolute_loss <loss>=0.789287663306\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15650.676012039185, \"sum\": 15650.676012039185, \"min\": 15650.676012039185}}, \"EndTime\": 1603775455.665588, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775440.014649}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #progress_metric: host=algo-1, completed 30 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 21043, \"sum\": 21043.0, \"min\": 21043}, \"Total Records Seen\": {\"count\": 1, \"max\": 4208246, \"sum\": 4208246.0, \"min\": 4208246}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 7, \"sum\": 7.0, \"min\": 7}}, \"EndTime\": 1603775455.665735, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 5}, \"StartTime\": 1603775440.014888}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44811.4675693 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=0 train rmse <loss>=0.980906432438\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=0 train mse <loss>=0.962177429199\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:55 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=0 train absolute_loss <loss>=0.778049468994\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:57 INFO 139876871620416] Iter[6] Batch [500]#011Speed: 44429.00 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:57 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=500 train rmse <loss>=0.993381054486\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:57 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=500 train mse <loss>=0.986805919411\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:10:57 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=500 train absolute_loss <loss>=0.781655235367\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:00 INFO 139876871620416] Iter[6] Batch [1000]#011Speed: 43204.89 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1000 train rmse <loss>=0.988850674277\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1000 train mse <loss>=0.977825656018\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1000 train absolute_loss <loss>=0.774971480089\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:02 INFO 139876871620416] Iter[6] Batch [1500]#011Speed: 45853.49 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1500 train rmse <loss>=0.986377094515\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1500 train mse <loss>=0.972939772584\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=1500 train absolute_loss <loss>=0.771254319884\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:04 INFO 139876871620416] Iter[6] Batch [2000]#011Speed: 45172.08 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:04 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2000 train rmse <loss>=0.984257373191\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:04 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2000 train mse <loss>=0.968762576681\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:04 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2000 train absolute_loss <loss>=0.769043577691\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:06 INFO 139876871620416] Iter[6] Batch [2500]#011Speed: 44962.23 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:06 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2500 train rmse <loss>=0.983053997491\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:06 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2500 train mse <loss>=0.966395161982\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:06 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=2500 train absolute_loss <loss>=0.767848946954\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:09 INFO 139876871620416] Iter[6] Batch [3000]#011Speed: 46020.66 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3000 train rmse <loss>=0.982806228038\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3000 train mse <loss>=0.965908081871\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3000 train absolute_loss <loss>=0.767227792978\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] Iter[6] Batch [3500]#011Speed: 44953.80 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3500 train rmse <loss>=0.981858332772\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3500 train mse <loss>=0.964045785633\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, batch=3500 train absolute_loss <loss>=0.766176105424\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:11:11.280] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 14, \"duration\": 15613, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, train rmse <loss>=0.98245268122\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, train mse <loss>=0.965213270837\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=6, train absolute_loss <loss>=0.766434397854\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15615.174055099487, \"sum\": 15615.174055099487, \"min\": 15615.174055099487}}, \"EndTime\": 1603775471.281087, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775455.665647}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #progress_metric: host=algo-1, completed 35 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 24550, \"sum\": 24550.0, \"min\": 24550}, \"Total Records Seen\": {\"count\": 1, \"max\": 4909587, \"sum\": 4909587.0, \"min\": 4909587}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 8, \"sum\": 8.0, \"min\": 8}}, \"EndTime\": 1603775471.281232, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 6}, \"StartTime\": 1603775455.665889}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44913.3520689 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=0 train rmse <loss>=0.958619735602\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=0 train mse <loss>=0.918951797485\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:11 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=0 train absolute_loss <loss>=0.758044052124\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:13 INFO 139876871620416] Iter[7] Batch [500]#011Speed: 44771.94 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:13 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=500 train rmse <loss>=0.968874123118\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:13 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=500 train mse <loss>=0.938717066447\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:13 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=500 train absolute_loss <loss>=0.758895815547\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:15 INFO 139876871620416] Iter[7] Batch [1000]#011Speed: 44149.02 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1000 train rmse <loss>=0.964159829587\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1000 train mse <loss>=0.929604176989\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:15 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1000 train absolute_loss <loss>=0.751828773491\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:18 INFO 139876871620416] Iter[7] Batch [1500]#011Speed: 44557.64 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:18 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1500 train rmse <loss>=0.961713327014\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:18 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1500 train mse <loss>=0.924892523357\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:18 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=1500 train absolute_loss <loss>=0.748055441016\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:20 INFO 139876871620416] Iter[7] Batch [2000]#011Speed: 44419.89 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:20 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2000 train rmse <loss>=0.959575532517\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:20 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2000 train mse <loss>=0.920785202606\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:20 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2000 train absolute_loss <loss>=0.745767496863\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:22 INFO 139876871620416] Iter[7] Batch [2500]#011Speed: 44012.52 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:22 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2500 train rmse <loss>=0.958365191206\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:22 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2500 train mse <loss>=0.918463839714\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:22 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=2500 train absolute_loss <loss>=0.744548608932\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:24 INFO 139876871620416] Iter[7] Batch [3000]#011Speed: 45211.57 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3000 train rmse <loss>=0.958122676218\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3000 train mse <loss>=0.917999062683\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3000 train absolute_loss <loss>=0.743898569506\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:26 INFO 139876871620416] Iter[7] Batch [3500]#011Speed: 45172.30 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3500 train rmse <loss>=0.957190266106\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3500 train mse <loss>=0.916213205529\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, batch=3500 train absolute_loss <loss>=0.742834864739\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:11:27.012] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 16, \"duration\": 15728, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, train rmse <loss>=0.957798253358\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, train mse <loss>=0.917377494135\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=7, train absolute_loss <loss>=0.743093962962\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15729.912042617798, \"sum\": 15729.912042617798, \"min\": 15729.912042617798}}, \"EndTime\": 1603775487.012938, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775471.281144}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #progress_metric: host=algo-1, completed 40 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 28057, \"sum\": 28057.0, \"min\": 28057}, \"Total Records Seen\": {\"count\": 1, \"max\": 5610928, \"sum\": 5610928.0, \"min\": 5610928}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 9, \"sum\": 9.0, \"min\": 9}}, \"EndTime\": 1603775487.013092, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 7}, \"StartTime\": 1603775471.283003}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44585.7113289 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=0 train rmse <loss>=0.934735017651\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=0 train mse <loss>=0.873729553223\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:27 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=0 train absolute_loss <loss>=0.736226196289\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:29 INFO 139876871620416] Iter[8] Batch [500]#011Speed: 44753.88 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:29 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=500 train rmse <loss>=0.944059723793\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:29 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=500 train mse <loss>=0.891248762089\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:29 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=500 train absolute_loss <loss>=0.735434828737\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:31 INFO 139876871620416] Iter[8] Batch [1000]#011Speed: 44311.14 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:31 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1000 train rmse <loss>=0.939201357043\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:31 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1000 train mse <loss>=0.882099189072\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:31 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1000 train absolute_loss <loss>=0.728097524348\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:33 INFO 139876871620416] Iter[8] Batch [1500]#011Speed: 43669.10 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1500 train rmse <loss>=0.93679579924\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1500 train mse <loss>=0.877586369473\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=1500 train absolute_loss <loss>=0.724324056891\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:35 INFO 139876871620416] Iter[8] Batch [2000]#011Speed: 45713.95 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2000 train rmse <loss>=0.934646902781\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2000 train mse <loss>=0.873564832879\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2000 train absolute_loss <loss>=0.721977423268\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:38 INFO 139876871620416] Iter[8] Batch [2500]#011Speed: 44682.72 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:38 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2500 train rmse <loss>=0.933429260586\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:38 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2500 train mse <loss>=0.871290184518\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:38 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=2500 train absolute_loss <loss>=0.720736079165\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:40 INFO 139876871620416] Iter[8] Batch [3000]#011Speed: 44091.79 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3000 train rmse <loss>=0.933188985474\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3000 train mse <loss>=0.870841682609\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3000 train absolute_loss <loss>=0.720061110827\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] Iter[8] Batch [3500]#011Speed: 45783.52 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3500 train rmse <loss>=0.93227120236\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3500 train mse <loss>=0.86912959475\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, batch=3500 train absolute_loss <loss>=0.718995762988\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:11:42.707] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 18, \"duration\": 15692, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, train rmse <loss>=0.932893900124\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, train mse <loss>=0.870291028889\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=8, train absolute_loss <loss>=0.719255810007\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15694.197177886963, \"sum\": 15694.197177886963, \"min\": 15694.197177886963}}, \"EndTime\": 1603775502.70747, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775487.012993}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #progress_metric: host=algo-1, completed 45 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 31564, \"sum\": 31564.0, \"min\": 31564}, \"Total Records Seen\": {\"count\": 1, \"max\": 6312269, \"sum\": 6312269.0, \"min\": 6312269}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 10, \"sum\": 10.0, \"min\": 10}}, \"EndTime\": 1603775502.707618, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 8}, \"StartTime\": 1603775487.013249}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44687.2094827 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=0 train rmse <loss>=0.909426217751\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=0 train mse <loss>=0.827056045532\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=0 train absolute_loss <loss>=0.713131561279\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:44 INFO 139876871620416] Iter[9] Batch [500]#011Speed: 44621.66 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=500 train rmse <loss>=0.918904702689\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=500 train mse <loss>=0.844385852623\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=500 train absolute_loss <loss>=0.711362638645\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:47 INFO 139876871620416] Iter[9] Batch [1000]#011Speed: 44609.70 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:47 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1000 train rmse <loss>=0.913949077711\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:47 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1000 train mse <loss>=0.83530291665\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:47 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1000 train absolute_loss <loss>=0.703814527186\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:49 INFO 139876871620416] Iter[9] Batch [1500]#011Speed: 42953.06 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1500 train rmse <loss>=0.911601394073\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1500 train mse <loss>=0.831017101677\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=1500 train absolute_loss <loss>=0.700069083303\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:51 INFO 139876871620416] Iter[9] Batch [2000]#011Speed: 43424.51 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2000 train rmse <loss>=0.909457379123\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2000 train mse <loss>=0.827112724441\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2000 train absolute_loss <loss>=0.697684329813\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:54 INFO 139876871620416] Iter[9] Batch [2500]#011Speed: 45654.28 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2500 train rmse <loss>=0.908240160833\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2500 train mse <loss>=0.82490018975\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:54 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=2500 train absolute_loss <loss>=0.696425826261\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:56 INFO 139876871620416] Iter[9] Batch [3000]#011Speed: 46032.29 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3000 train rmse <loss>=0.908005635475\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3000 train mse <loss>=0.824474234055\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3000 train absolute_loss <loss>=0.695728376847\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] Iter[9] Batch [3500]#011Speed: 46070.51 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3500 train rmse <loss>=0.907109075784\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3500 train mse <loss>=0.822846875369\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, batch=3500 train absolute_loss <loss>=0.694671671523\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:11:58.389] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 20, \"duration\": 15680, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, train rmse <loss>=0.907747640015\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, train mse <loss>=0.824005777953\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=9, train absolute_loss <loss>=0.694932571294\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15682.078838348389, \"sum\": 15682.078838348389, \"min\": 15682.078838348389}}, \"EndTime\": 1603775518.389873, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775502.707529}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #progress_metric: host=algo-1, completed 50 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 35071, \"sum\": 35071.0, \"min\": 35071}, \"Total Records Seen\": {\"count\": 1, \"max\": 7013610, \"sum\": 7013610.0, \"min\": 7013610}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 11, \"sum\": 11.0, \"min\": 11}}, \"EndTime\": 1603775518.390014, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 9}, \"StartTime\": 1603775502.70777}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44721.7573309 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=0 train rmse <loss>=0.883146283001\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=0 train mse <loss>=0.779947357178\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:11:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=0 train absolute_loss <loss>=0.688646240234\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:00 INFO 139876871620416] Iter[10] Batch [500]#011Speed: 42712.81 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=500 train rmse <loss>=0.893519762344\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=500 train mse <loss>=0.7983775657\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=500 train absolute_loss <loss>=0.686765473417\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:03 INFO 139876871620416] Iter[10] Batch [1000]#011Speed: 43423.25 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1000 train rmse <loss>=0.888514379081\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1000 train mse <loss>=0.789457801834\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1000 train absolute_loss <loss>=0.679084613811\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:05 INFO 139876871620416] Iter[10] Batch [1500]#011Speed: 43451.41 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1500 train rmse <loss>=0.886239943483\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1500 train mse <loss>=0.785421237425\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=1500 train absolute_loss <loss>=0.675398259401\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:07 INFO 139876871620416] Iter[10] Batch [2000]#011Speed: 41371.75 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2000 train rmse <loss>=0.884122146556\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2000 train mse <loss>=0.78167197003\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2000 train absolute_loss <loss>=0.673019147279\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:10 INFO 139876871620416] Iter[10] Batch [2500]#011Speed: 41470.01 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2500 train rmse <loss>=0.882918389405\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2500 train mse <loss>=0.779544882349\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:10 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=2500 train absolute_loss <loss>=0.671759746541\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:12 INFO 139876871620416] Iter[10] Batch [3000]#011Speed: 43167.37 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3000 train rmse <loss>=0.882697509464\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3000 train mse <loss>=0.779154893213\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3000 train absolute_loss <loss>=0.671067018874\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] Iter[10] Batch [3500]#011Speed: 41929.85 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3500 train rmse <loss>=0.88183252911\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3500 train mse <loss>=0.777628609397\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, batch=3500 train absolute_loss <loss>=0.670034065181\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:12:14.904] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 22, \"duration\": 16510, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, train rmse <loss>=0.882488130708\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, train mse <loss>=0.778785300841\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=10, train absolute_loss <loss>=0.670295814621\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16512.53604888916, \"sum\": 16512.53604888916, \"min\": 16512.53604888916}}, \"EndTime\": 1603775534.904785, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775518.389926}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #progress_metric: host=algo-1, completed 55 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 38578, \"sum\": 38578.0, \"min\": 38578}, \"Total Records Seen\": {\"count\": 1, \"max\": 7714951, \"sum\": 7714951.0, \"min\": 7714951}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 12, \"sum\": 12.0, \"min\": 12}}, \"EndTime\": 1603775534.904933, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 10}, \"StartTime\": 1603775518.392225}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=42472.5821229 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=0 train rmse <loss>=0.856578769355\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=0 train mse <loss>=0.73372718811\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=0 train absolute_loss <loss>=0.663539733887\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:17 INFO 139876871620416] Iter[11] Batch [500]#011Speed: 44964.03 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=500 train rmse <loss>=0.86810407753\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=500 train mse <loss>=0.753604689425\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=500 train absolute_loss <loss>=0.661993781906\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:19 INFO 139876871620416] Iter[11] Batch [1000]#011Speed: 44517.14 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1000 train rmse <loss>=0.863092099946\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1000 train mse <loss>=0.74492797299\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1000 train absolute_loss <loss>=0.654286700505\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:21 INFO 139876871620416] Iter[11] Batch [1500]#011Speed: 44648.36 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1500 train rmse <loss>=0.860902604642\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1500 train mse <loss>=0.74115329468\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=1500 train absolute_loss <loss>=0.650698833529\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:23 INFO 139876871620416] Iter[11] Batch [2000]#011Speed: 44979.58 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2000 train rmse <loss>=0.858832359035\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2000 train mse <loss>=0.737593020926\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2000 train absolute_loss <loss>=0.648370069936\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:26 INFO 139876871620416] Iter[11] Batch [2500]#011Speed: 45123.32 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2500 train rmse <loss>=0.857654826694\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2500 train mse <loss>=0.735571801751\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=2500 train absolute_loss <loss>=0.647149036305\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:28 INFO 139876871620416] Iter[11] Batch [3000]#011Speed: 45049.86 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3000 train rmse <loss>=0.857455175891\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3000 train mse <loss>=0.735229378662\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3000 train absolute_loss <loss>=0.64648543376\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] Iter[11] Batch [3500]#011Speed: 44644.05 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3500 train rmse <loss>=0.856631114809\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3500 train mse <loss>=0.73381686686\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, batch=3500 train absolute_loss <loss>=0.645503397781\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:12:30.552] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 24, \"duration\": 15646, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, train rmse <loss>=0.857304829696\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, train mse <loss>=0.73497157102\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=11, train absolute_loss <loss>=0.645765643309\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15647.977113723755, \"sum\": 15647.977113723755, \"min\": 15647.977113723755}}, \"EndTime\": 1603775550.553093, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775534.904842}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #progress_metric: host=algo-1, completed 60 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 42085, \"sum\": 42085.0, \"min\": 42085}, \"Total Records Seen\": {\"count\": 1, \"max\": 8416292, \"sum\": 8416292.0, \"min\": 8416292}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 13, \"sum\": 13.0, \"min\": 13}}, \"EndTime\": 1603775550.553233, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 11}, \"StartTime\": 1603775534.905094}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44819.2250225 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=0 train rmse <loss>=0.830425570667\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=0 train mse <loss>=0.689606628418\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=0 train absolute_loss <loss>=0.637864265442\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:32 INFO 139876871620416] Iter[12] Batch [500]#011Speed: 42178.87 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=500 train rmse <loss>=0.842860188704\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=500 train mse <loss>=0.710413297703\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=500 train absolute_loss <loss>=0.637525669334\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:35 INFO 139876871620416] Iter[12] Batch [1000]#011Speed: 42680.58 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1000 train rmse <loss>=0.837872611561\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1000 train mse <loss>=0.702030513203\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1000 train absolute_loss <loss>=0.629899294241\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:37 INFO 139876871620416] Iter[12] Batch [1500]#011Speed: 43752.55 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1500 train rmse <loss>=0.835773436805\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1500 train mse <loss>=0.698517237669\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=1500 train absolute_loss <loss>=0.626447379553\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:39 INFO 139876871620416] Iter[12] Batch [2000]#011Speed: 43229.56 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2000 train rmse <loss>=0.833767337186\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2000 train mse <loss>=0.695167972559\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2000 train absolute_loss <loss>=0.62417984472\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:42 INFO 139876871620416] Iter[12] Batch [2500]#011Speed: 44471.13 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2500 train rmse <loss>=0.832623640817\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2500 train mse <loss>=0.693262127247\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=2500 train absolute_loss <loss>=0.623016034461\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:44 INFO 139876871620416] Iter[12] Batch [3000]#011Speed: 44837.26 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3000 train rmse <loss>=0.832448995358\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3000 train mse <loss>=0.692971329872\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3000 train absolute_loss <loss>=0.622395277853\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] Iter[12] Batch [3500]#011Speed: 43011.99 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3500 train rmse <loss>=0.831670893865\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3500 train mse <loss>=0.691676475701\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, batch=3500 train absolute_loss <loss>=0.621477884305\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:12:46.709] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 26, \"duration\": 16154, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, train rmse <loss>=0.832363674362\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, train mse <loss>=0.692829286397\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=12, train absolute_loss <loss>=0.621740870099\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16156.224012374878, \"sum\": 16156.224012374878, \"min\": 16156.224012374878}}, \"EndTime\": 1603775566.70963, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775550.553146}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #progress_metric: host=algo-1, completed 65 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 45592, \"sum\": 45592.0, \"min\": 45592}, \"Total Records Seen\": {\"count\": 1, \"max\": 9117633, \"sum\": 9117633.0, \"min\": 9117633}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 14, \"sum\": 14.0, \"min\": 14}}, \"EndTime\": 1603775566.709779, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 12}, \"StartTime\": 1603775550.553384}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43409.274767 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=0 train rmse <loss>=0.805160424969\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=0 train mse <loss>=0.648283309937\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=0 train absolute_loss <loss>=0.612690124512\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:49 INFO 139876871620416] Iter[13] Batch [500]#011Speed: 42924.94 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=500 train rmse <loss>=0.81792500922\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=500 train mse <loss>=0.669001320708\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=500 train absolute_loss <loss>=0.613578565754\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:51 INFO 139876871620416] Iter[13] Batch [1000]#011Speed: 41978.27 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1000 train rmse <loss>=0.812981568827\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1000 train mse <loss>=0.660939031252\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1000 train absolute_loss <loss>=0.606073993908\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:53 INFO 139876871620416] Iter[13] Batch [1500]#011Speed: 42846.13 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1500 train rmse <loss>=0.810974329997\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1500 train mse <loss>=0.657679363914\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=1500 train absolute_loss <loss>=0.602750070001\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:56 INFO 139876871620416] Iter[13] Batch [2000]#011Speed: 44205.46 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2000 train rmse <loss>=0.809044586963\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2000 train mse <loss>=0.654553143693\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2000 train absolute_loss <loss>=0.600546902109\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:58 INFO 139876871620416] Iter[13] Batch [2500]#011Speed: 44652.56 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2500 train rmse <loss>=0.807937537073\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2500 train mse <loss>=0.652763063811\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:12:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=2500 train absolute_loss <loss>=0.599427783879\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:00 INFO 139876871620416] Iter[13] Batch [3000]#011Speed: 43257.76 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3000 train rmse <loss>=0.807789278371\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3000 train mse <loss>=0.652523518251\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3000 train absolute_loss <loss>=0.598843292979\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] Iter[13] Batch [3500]#011Speed: 43628.47 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3500 train rmse <loss>=0.807058523096\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3500 train mse <loss>=0.651343459702\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, batch=3500 train absolute_loss <loss>=0.597991474939\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:13:02.895] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 28, \"duration\": 16184, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, train rmse <loss>=0.807771252803\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, train mse <loss>=0.652494396855\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=13, train absolute_loss <loss>=0.5982555951\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16186.234951019287, \"sum\": 16186.234951019287, \"min\": 16186.234951019287}}, \"EndTime\": 1603775582.896205, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775566.709684}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #progress_metric: host=algo-1, completed 70 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 49099, \"sum\": 49099.0, \"min\": 49099}, \"Total Records Seen\": {\"count\": 1, \"max\": 9818974, \"sum\": 9818974.0, \"min\": 9818974}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 15, \"sum\": 15.0, \"min\": 15}}, \"EndTime\": 1603775582.896349, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 13}, \"StartTime\": 1603775566.709946}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43328.790786 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=0 train rmse <loss>=0.780935874348\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=0 train mse <loss>=0.609860839844\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:02 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=0 train absolute_loss <loss>=0.588535194397\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:05 INFO 139876871620416] Iter[14] Batch [500]#011Speed: 43460.42 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=500 train rmse <loss>=0.793377455526\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=500 train mse <loss>=0.629447786937\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=500 train absolute_loss <loss>=0.590134072675\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:07 INFO 139876871620416] Iter[14] Batch [1000]#011Speed: 42143.28 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1000 train rmse <loss>=0.788493261872\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1000 train mse <loss>=0.621721624018\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1000 train absolute_loss <loss>=0.582753704082\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:09 INFO 139876871620416] Iter[14] Batch [1500]#011Speed: 43333.45 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1500 train rmse <loss>=0.786580507881\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1500 train mse <loss>=0.618708895379\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=1500 train absolute_loss <loss>=0.579549003515\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:12 INFO 139876871620416] Iter[14] Batch [2000]#011Speed: 41613.70 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2000 train rmse <loss>=0.78473709132\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2000 train mse <loss>=0.615812302494\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2000 train absolute_loss <loss>=0.577414698298\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:14 INFO 139876871620416] Iter[14] Batch [2500]#011Speed: 40728.19 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2500 train rmse <loss>=0.783667834442\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2500 train mse <loss>=0.614135274738\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=2500 train absolute_loss <loss>=0.576338021646\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:17 INFO 139876871620416] Iter[14] Batch [3000]#011Speed: 41183.74 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3000 train rmse <loss>=0.783547600374\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3000 train mse <loss>=0.613946842053\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:17 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3000 train absolute_loss <loss>=0.575784737874\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] Iter[14] Batch [3500]#011Speed: 44970.72 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3500 train rmse <loss>=0.782864087989\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3500 train mse <loss>=0.612876180264\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, batch=3500 train absolute_loss <loss>=0.574988489562\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:13:19.427] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 30, \"duration\": 16529, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, train rmse <loss>=0.783597644987\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, train mse <loss>=0.614025269229\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=14, train absolute_loss <loss>=0.57525386298\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16530.82585334778, \"sum\": 16530.82585334778, \"min\": 16530.82585334778}}, \"EndTime\": 1603775599.427347, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775582.896261}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #progress_metric: host=algo-1, completed 75 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 52606, \"sum\": 52606.0, \"min\": 52606}, \"Total Records Seen\": {\"count\": 1, \"max\": 10520315, \"sum\": 10520315.0, \"min\": 10520315}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 16, \"sum\": 16.0, \"min\": 16}}, \"EndTime\": 1603775599.427468, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 14}, \"StartTime\": 1603775582.896499}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=42425.7059716 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=0 train rmse <loss>=0.757697720045\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=0 train mse <loss>=0.574105834961\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=0 train absolute_loss <loss>=0.565243186951\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:21 INFO 139876871620416] Iter[15] Batch [500]#011Speed: 44448.51 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=500 train rmse <loss>=0.769278756331\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=500 train mse <loss>=0.591789804942\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=500 train absolute_loss <loss>=0.567203118406\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:24 INFO 139876871620416] Iter[15] Batch [1000]#011Speed: 42226.04 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1000 train rmse <loss>=0.764467665045\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1000 train mse <loss>=0.584410810899\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:24 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1000 train absolute_loss <loss>=0.559936331064\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:26 INFO 139876871620416] Iter[15] Batch [1500]#011Speed: 42519.67 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1500 train rmse <loss>=0.762654293424\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1500 train mse <loss>=0.581641571279\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:26 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=1500 train absolute_loss <loss>=0.556847005866\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:28 INFO 139876871620416] Iter[15] Batch [2000]#011Speed: 43594.24 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2000 train rmse <loss>=0.760905416738\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2000 train mse <loss>=0.578977053221\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2000 train absolute_loss <loss>=0.554800110683\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:30 INFO 139876871620416] Iter[15] Batch [2500]#011Speed: 44979.95 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2500 train rmse <loss>=0.759875341986\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2500 train mse <loss>=0.577410535359\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=2500 train absolute_loss <loss>=0.553764937888\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:33 INFO 139876871620416] Iter[15] Batch [3000]#011Speed: 43913.14 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3000 train rmse <loss>=0.759785648162\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3000 train mse <loss>=0.577274231153\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:33 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3000 train absolute_loss <loss>=0.553248499585\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] Iter[15] Batch [3500]#011Speed: 45157.76 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3500 train rmse <loss>=0.759149033838\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3500 train mse <loss>=0.576307255577\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, batch=3500 train absolute_loss <loss>=0.552504753346\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:13:35.441] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 32, \"duration\": 16012, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, train rmse <loss>=0.7599043182\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, train mse <loss>=0.577454572818\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=15, train absolute_loss <loss>=0.552770980387\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16014.396905899048, \"sum\": 16014.396905899048, \"min\": 16014.396905899048}}, \"EndTime\": 1603775615.44202, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775599.427394}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #progress_metric: host=algo-1, completed 80 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 56113, \"sum\": 56113.0, \"min\": 56113}, \"Total Records Seen\": {\"count\": 1, \"max\": 11221656, \"sum\": 11221656.0, \"min\": 11221656}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 17, \"sum\": 17.0, \"min\": 17}}, \"EndTime\": 1603775615.442162, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 15}, \"StartTime\": 1603775599.427598}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43793.7461887 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=0 train rmse <loss>=0.735362767594\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=0 train mse <loss>=0.540758399963\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:35 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=0 train absolute_loss <loss>=0.54219291687\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:37 INFO 139876871620416] Iter[16] Batch [500]#011Speed: 41253.34 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=500 train rmse <loss>=0.745690715247\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=500 train mse <loss>=0.556054642805\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=500 train absolute_loss <loss>=0.544854216966\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:40 INFO 139876871620416] Iter[16] Batch [1000]#011Speed: 43001.84 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1000 train rmse <loss>=0.740964676423\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1000 train mse <loss>=0.549028651707\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:40 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1000 train absolute_loss <loss>=0.537730387441\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:42 INFO 139876871620416] Iter[16] Batch [1500]#011Speed: 45184.10 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1500 train rmse <loss>=0.739256393297\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1500 train mse <loss>=0.54650001503\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:42 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=1500 train absolute_loss <loss>=0.534766308697\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:44 INFO 139876871620416] Iter[16] Batch [2000]#011Speed: 45261.97 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2000 train rmse <loss>=0.737607863423\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2000 train mse <loss>=0.544065360184\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:44 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2000 train absolute_loss <loss>=0.532812871995\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:46 INFO 139876871620416] Iter[16] Batch [2500]#011Speed: 43637.76 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2500 train rmse <loss>=0.736618668287\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2500 train mse <loss>=0.542607062468\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:46 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=2500 train absolute_loss <loss>=0.531819119047\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:49 INFO 139876871620416] Iter[16] Batch [3000]#011Speed: 43930.65 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3000 train rmse <loss>=0.736562135327\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3000 train mse <loss>=0.542523779197\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:49 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3000 train absolute_loss <loss>=0.531347798618\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] Iter[16] Batch [3500]#011Speed: 43659.98 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3500 train rmse <loss>=0.735972057876\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3500 train mse <loss>=0.541654869974\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, batch=3500 train absolute_loss <loss>=0.530654240989\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:13:51.509] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 34, \"duration\": 16066, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, train rmse <loss>=0.736749980758\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, train mse <loss>=0.542800534147\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=16, train absolute_loss <loss>=0.53092118976\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16068.025827407837, \"sum\": 16068.025827407837, \"min\": 16068.025827407837}}, \"EndTime\": 1603775631.51036, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775615.442074}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #progress_metric: host=algo-1, completed 85 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 59620, \"sum\": 59620.0, \"min\": 59620}, \"Total Records Seen\": {\"count\": 1, \"max\": 11922997, \"sum\": 11922997.0, \"min\": 11922997}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 18, \"sum\": 18.0, \"min\": 18}}, \"EndTime\": 1603775631.510505, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 16}, \"StartTime\": 1603775615.442311}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43647.5628357 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=0 train rmse <loss>=0.713903630653\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=0 train mse <loss>=0.50965839386\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:51 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=0 train absolute_loss <loss>=0.520324859619\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:53 INFO 139876871620416] Iter[17] Batch [500]#011Speed: 42093.09 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=500 train rmse <loss>=0.722671253489\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=500 train mse <loss>=0.522253740619\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:53 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=500 train absolute_loss <loss>=0.523166272207\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:56 INFO 139876871620416] Iter[17] Batch [1000]#011Speed: 43595.09 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1000 train rmse <loss>=0.718039330045\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1000 train mse <loss>=0.515580479491\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:56 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1000 train absolute_loss <loss>=0.516217428164\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:58 INFO 139876871620416] Iter[17] Batch [1500]#011Speed: 43476.01 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1500 train rmse <loss>=0.716440588212\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1500 train mse <loss>=0.513287116438\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:13:58 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=1500 train absolute_loss <loss>=0.513393546377\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:00 INFO 139876871620416] Iter[17] Batch [2000]#011Speed: 43395.73 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2000 train rmse <loss>=0.714895268354\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2000 train mse <loss>=0.511075244715\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:00 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2000 train absolute_loss <loss>=0.511550076669\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:03 INFO 139876871620416] Iter[17] Batch [2500]#011Speed: 43272.36 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2500 train rmse <loss>=0.713948306724\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2500 train mse <loss>=0.509722184673\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:03 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=2500 train absolute_loss <loss>=0.510602286277\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:05 INFO 139876871620416] Iter[17] Batch [3000]#011Speed: 44385.86 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3000 train rmse <loss>=0.713926859802\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3000 train mse <loss>=0.509691561146\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:05 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3000 train absolute_loss <loss>=0.510175412161\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] Iter[17] Batch [3500]#011Speed: 44958.25 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3500 train rmse <loss>=0.713382767368\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3500 train mse <loss>=0.508914972778\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, batch=3500 train absolute_loss <loss>=0.509531990875\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:14:07.607] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 36, \"duration\": 16095, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, train rmse <loss>=0.714184223324\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, train mse <loss>=0.510059104845\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=17, train absolute_loss <loss>=0.50979965012\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 16097.228050231934, \"sum\": 16097.228050231934, \"min\": 16097.228050231934}}, \"EndTime\": 1603775647.607909, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775631.510416}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #progress_metric: host=algo-1, completed 90 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 63127, \"sum\": 63127.0, \"min\": 63127}, \"Total Records Seen\": {\"count\": 1, \"max\": 12624338, \"sum\": 12624338.0, \"min\": 12624338}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 19, \"sum\": 19.0, \"min\": 19}}, \"EndTime\": 1603775647.608025, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 17}, \"StartTime\": 1603775631.510658}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43568.4881858 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=0 train rmse <loss>=0.693327243974\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=0 train mse <loss>=0.480702667236\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:07 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=0 train absolute_loss <loss>=0.498664627075\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:09 INFO 139876871620416] Iter[18] Batch [500]#011Speed: 43822.57 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=500 train rmse <loss>=0.70026552186\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=500 train mse <loss>=0.490371801106\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:09 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=500 train absolute_loss <loss>=0.502253253883\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:12 INFO 139876871620416] Iter[18] Batch [1000]#011Speed: 43550.38 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1000 train rmse <loss>=0.695734007214\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1000 train mse <loss>=0.484045808794\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:12 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1000 train absolute_loss <loss>=0.495496676595\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:14 INFO 139876871620416] Iter[18] Batch [1500]#011Speed: 44357.35 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1500 train rmse <loss>=0.694246715687\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1500 train mse <loss>=0.481978502242\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:14 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=1500 train absolute_loss <loss>=0.492813833263\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:16 INFO 139876871620416] Iter[18] Batch [2000]#011Speed: 44056.24 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:16 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2000 train rmse <loss>=0.69280449293\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:16 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2000 train mse <loss>=0.479978065424\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:16 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2000 train absolute_loss <loss>=0.491078932332\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:18 INFO 139876871620416] Iter[18] Batch [2500]#011Speed: 43829.45 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2500 train rmse <loss>=0.691900231898\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2500 train mse <loss>=0.4787259309\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:19 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=2500 train absolute_loss <loss>=0.490174570644\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:21 INFO 139876871620416] Iter[18] Batch [3000]#011Speed: 44281.37 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3000 train rmse <loss>=0.691914759577\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3000 train mse <loss>=0.47874603452\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:21 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3000 train absolute_loss <loss>=0.489792442106\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] Iter[18] Batch [3500]#011Speed: 44650.81 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3500 train rmse <loss>=0.691415743571\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3500 train mse <loss>=0.478055730458\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, batch=3500 train absolute_loss <loss>=0.48920047298\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:14:23.524] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 38, \"duration\": 15914, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, train rmse <loss>=0.692241593779\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, train mse <loss>=0.479198424157\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=18, train absolute_loss <loss>=0.489468764356\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15916.598796844482, \"sum\": 15916.598796844482, \"min\": 15916.598796844482}}, \"EndTime\": 1603775663.524773, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775647.607953}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #progress_metric: host=algo-1, completed 95 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 66634, \"sum\": 66634.0, \"min\": 66634}, \"Total Records Seen\": {\"count\": 1, \"max\": 13325679, \"sum\": 13325679.0, \"min\": 13325679}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 20, \"sum\": 20.0, \"min\": 20}}, \"EndTime\": 1603775663.524927, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 18}, \"StartTime\": 1603775647.608152}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=44062.7932015 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=0 train rmse <loss>=0.673624034989\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=0 train mse <loss>=0.453769340515\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:23 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=0 train absolute_loss <loss>=0.477600135803\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:25 INFO 139876871620416] Iter[19] Batch [500]#011Speed: 42946.34 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=500 train rmse <loss>=0.6785015498\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=500 train mse <loss>=0.460364353081\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:25 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=500 train absolute_loss <loss>=0.482141896779\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:28 INFO 139876871620416] Iter[19] Batch [1000]#011Speed: 43052.10 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1000 train rmse <loss>=0.674074751701\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1000 train mse <loss>=0.454376770881\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:28 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1000 train absolute_loss <loss>=0.475591604631\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:30 INFO 139876871620416] Iter[19] Batch [1500]#011Speed: 44543.51 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1500 train rmse <loss>=0.672698010724\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1500 train mse <loss>=0.452522613632\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:30 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=1500 train absolute_loss <loss>=0.473057397516\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:32 INFO 139876871620416] Iter[19] Batch [2000]#011Speed: 44041.01 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2000 train rmse <loss>=0.671356155115\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2000 train mse <loss>=0.450719087011\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:32 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2000 train absolute_loss <loss>=0.471430213658\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:34 INFO 139876871620416] Iter[19] Batch [2500]#011Speed: 46113.08 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2500 train rmse <loss>=0.670494068096\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2500 train mse <loss>=0.449562295352\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:34 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=2500 train absolute_loss <loss>=0.470565654796\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:37 INFO 139876871620416] Iter[19] Batch [3000]#011Speed: 43732.73 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3000 train rmse <loss>=0.670544502702\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3000 train mse <loss>=0.449629930104\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:37 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3000 train absolute_loss <loss>=0.470229934812\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] Iter[19] Batch [3500]#011Speed: 42949.00 samples/sec\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3500 train rmse <loss>=0.670089284493\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3500 train mse <loss>=0.449019649193\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, batch=3500 train absolute_loss <loss>=0.469689459334\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:14:39.509] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/train\", \"epoch\": 40, \"duration\": 15983, \"num_examples\": 3507, \"num_bytes\": 46811980}\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, train rmse <loss>=0.670940354508\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, train mse <loss>=0.450160959307\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, epoch=19, train absolute_loss <loss>=0.469958699077\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, train rmse <loss>=0.670940354508\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, train mse <loss>=0.450160959307\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #quality_metric: host=algo-1, train absolute_loss <loss>=0.469958699077\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 15985.281944274902, \"sum\": 15985.281944274902, \"min\": 15985.281944274902}}, \"EndTime\": 1603775679.510396, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775663.524826}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #progress_metric: host=algo-1, completed 100 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 3507, \"sum\": 3507.0, \"min\": 3507}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Total Batches Seen\": {\"count\": 1, \"max\": 70141, \"sum\": 70141.0, \"min\": 70141}, \"Total Records Seen\": {\"count\": 1, \"max\": 14027020, \"sum\": 14027020.0, \"min\": 14027020}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 701341, \"sum\": 701341.0, \"min\": 701341}, \"Reset Count\": {\"count\": 1, \"max\": 21, \"sum\": 21.0, \"min\": 21}}, \"EndTime\": 1603775679.510535, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 19}, \"StartTime\": 1603775663.525092}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] #throughput_metric: host=algo-1, train throughput=43873.4601088 records/second\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 WARNING 139876871620416] wait_for_all_workers will not sync workers since the kv store is not running distributed\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:39 INFO 139876871620416] Pulling entire model from kvstore to finalize\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"finalize.time\": {\"count\": 1, \"max\": 22.823095321655273, \"sum\": 22.823095321655273, \"min\": 22.823095321655273}}, \"EndTime\": 1603775679.533584, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775679.510449}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:14:40 INFO 139876871620416] Saved checkpoint to \"/tmp/tmp9ljLpC/state-0001.params\"\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:14:41.210] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/test\", \"epoch\": 0, \"duration\": 321681, \"num_examples\": 1, \"num_bytes\": 13344}\u001b[0m\n",
      "\u001b[34m[2020-10-27 05:16:05.118] [tensorio] [info] epoch_stats={\"data_pipeline\": \"/opt/ml/input/data/test\", \"epoch\": 1, \"duration\": 83908, \"num_examples\": 1169, \"num_bytes\": 15603920}\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1169, \"sum\": 1169.0, \"min\": 1169}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 1169, \"sum\": 1169.0, \"min\": 1169}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 233781, \"sum\": 233781.0, \"min\": 233781}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1169, \"sum\": 1169.0, \"min\": 1169}, \"Total Records Seen\": {\"count\": 1, \"max\": 233781, \"sum\": 233781.0, \"min\": 233781}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 233781, \"sum\": 233781.0, \"min\": 233781}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1603775765.118634, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"test_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775681.209957}\n",
      "\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #test_score (algo-1) : ('rmse', 1.0610274393348178)\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #test_score (algo-1) : ('mse', 1.1257792270214004)\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #test_score (algo-1) : ('absolute_loss', 0.8683851257575115)\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #quality_metric: host=algo-1, test rmse <loss>=1.06102743933\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #quality_metric: host=algo-1, test mse <loss>=1.12577922702\u001b[0m\n",
      "\u001b[34m[10/27/2020 05:16:05 INFO 139876871620416] #quality_metric: host=algo-1, test absolute_loss <loss>=0.868385125758\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"totaltime\": {\"count\": 1, \"max\": 405638.02218437195, \"sum\": 405638.02218437195, \"min\": 405638.02218437195}, \"setuptime\": {\"count\": 1, \"max\": 40.73810577392578, \"sum\": 40.73810577392578, \"min\": 40.73810577392578}}, \"EndTime\": 1603775765.119993, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1603775679.533749}\n",
      "\u001b[0m\n",
      "Training seconds: 511\n",
      "Billable seconds: 511\n"
     ]
    }
   ],
   "source": [
    "fm = sagemaker.estimator.Estimator.attach(fm_best_model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can re-run the cells in Batch Inference and Evaluation section to evaluate the performance of the model with tuned hyper-parameters. \n",
    "\n",
    "Assuming batch inference is carried out, let's calculate predictions for our test dataset and see if we do better than the training job with default hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('MSE:', 1.1478783285179637)\n"
     ]
    }
   ],
   "source": [
    "print('MSE:', np.mean((test_Y - test_preds) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Wrap-up\n",
    "\n",
    "In this example, we developed a deep learning model to predict customer ratings.  This could serve as the foundation of a recommender system in a variety of use cases.  However, there are many ways in which it could be improved.  For example we did very little with:\n",
    "- hyperparameter tuning\n",
    "- controlling for overfitting (early stopping, dropout, etc.)\n",
    "- testing whether binarizing our target variable would improve results\n",
    "- including other information sources (video genres, historical ratings, time of review)\n",
    "- adjusting our threshold for user and item inclusion \n",
    "\n",
    "In addition to improving the model, we could improve the engineering by:\n",
    "- Setting the context and key value store up for distributed training\n",
    "- Fine tuning our data ingestion (e.g. num_workers on our data iterators) to ensure we're fully utilizing our GPU\n",
    "- Thinking about how pre-processing would need to change as datasets scale beyond a single machine\n",
    "\n",
    "Beyond that, recommenders are a very active area of research and techniques from active learning, reinforcement learning, segmentation, ensembling, and more should be investigated to deliver well-rounded recommendations.\n",
    "\n",
    "### Clean-up (optional)\n",
    "\n",
    "Let's finish by deleting our endpoint to avoid stray hosting charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name_contains = ['-fm-', 'factorization-machines-']\n",
    "for name in endpoint_name_contains:\n",
    "    endpoints = sm.list_endpoints(NameContains=name, StatusEquals='InService')\n",
    "    endpoint_names = [r['EndpointName'] for r in endpoints['Endpoints']]\n",
    "    for endpoint_name in endpoint_names:\n",
    "        print(\"Deleting endpoint: \" + endpoint_name)\n",
    "        sm.delete_endpoint(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python2",
   "language": "python",
   "name": "conda_python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.17"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
